Introduction to Machine Learning

From Math to Code

Ruye Wang

Contents

List of illustrations page viii List of tables ix List of boxes x List of contributors xi Notation xii

Part I Mathematical Fundations 1

1  Solving Equations  3

1.1 Linear Equation Systems 4 1.2 The Bisection and Secant methods 6 1.3 Fixed point iteration 10 1.4 Newton-Raphson Method (Univariate) 20 1.5 Newton-Raphson Method (Multivariate) 26

2  Optimization  33

2.1 Unconstrained Optimization 34 2.1.1 Optimization by Solving Equations 34 2.1.2 Newton’s method 35 2.1.3 Gradient Descent Method 39 2.1.4 Line minimization 42 2.1.5 Quasi-Newton Methods 47 2.1.6 Conjugate gradient method 54 2.1.7 Issues of Local/Global Minimum 63

2.2 Constrained Optimization 64 2.2.1 Optimization with Equality Constraints 65 2.2.2 Optimization with Inequality Constraints 67 2.2.3 Duality and KKT Conditions 71 2.2.4 Linear Programming (LP) 74 2.2.5 The Simplex Algorithm 80 2.2.6 Quadratic Programming (QP) 86 2.2.7 Interior Point Methods 87 3.1 Univariate and Multivariate Random Variables 95

3.1.1 Univariate Variables 95 3.1.2 Multivariate random vectors 101 3.1.3 The Normal Distribution 105 3.1.4 Function of Random Variables 112

3.2 Entropy and Information 114 3.2.1 Entropy 114 3.2.2 Mutual Information and Kullback-Leibler Divergence 121 3.2.3 Entropy Maximization 123

3.3 Bayesian Inference 125 3.3.1 Bayes’ Theorem 125 3.3.2 Maximum Likelihood Estimation of Model Parameters 128 3.3.3 Natural gradient 130 3.3.4 Fisher Information and Natural Gradient 131

3.4 Bias-Variance Tradeoff 133 3.5 Principal Component Analysis (PCA) 134 3.6 Student’s t-test 137 3.7 Hypergeometric Distribution 139

Part II Regression 141

4  Linear Regression  143

4.1 Regression as Optimization 143 4.2 Linear Least Squares Regression 147 4.3 Bias-Variance Tradeoff and Ridge Regression 154 4.4 Regression Based on Basis Functions 158 4.5 Bayesian Linear Regression 162

5  Nonlinear Regression  169

5.1 Nonlinear Least Squares Regression 169 5.2 Parameter Estimation by Natural Gradient Descent 175

6  Logistic and Softmax Regression  177

6.1 Logistic Regression and Binary Classification 177 6.2 Softmax Regression for Multiclass Classification 183

7  Gaussian Process Regression and Classification  191

7.1 Gaussian Process Regression 191 7.2 Gaussian Process Classifier – Binary 196 7.3 Gaussian Process Classifier – Multi-Class 203

Part III Feature Selection 211

8  Feature Selection  213

8.1 Dimensionality Reduction 213 8.2 Distances and Separability Measurements 213

9  Principal Component Analysis  221

9.1 Covariance and Correlation 221 9.2 Karhunen-Loeve Transformation 223 9.3 Optimality of KLT 225 9.4 Geometric Interpretation of the KLT 228 9.5 Computation of the KLT 229 9.6 Comparison with Other Orthogonal Transforms 231 9.7 Application to Image Data 232

10  Variations of PCA  237

10.1 Kernel Methods 237 10.2 Kernel PCA 240 10.3 Factor Analysis and Expectation Maximization 243 10.4 Probabilistic PCA 249 10.5 Classical Multidimensional Scaling 253 10.6 t-Distributed Stochastic Neighbor Embedding (t-SNE) 257

11  Independent Component Analysis  262

11.1 Independence and Non-Gaussianity 262 11.2 Preprocessing 264 11.3 Non-Gaussianity Maximization 265 11.4 The FastICA Algorithm 266 11.5 Likelihood Maximization 269

Part IV Classification 273

12  Statistic Classification  275

12.1 Discriminative vs. Generative Methods for Classification 275 12.2 K Nearest Neighbor and Minimum Distance Classifiers 277 12.3 Naive Bayes Classification 279 12.4 AdaBoost 286

13  Support Vector machine  295

13.1 Maximal Margin and Support Vectors 295 13.2 Kernel Mapping 301 13.3 Soft Margin SVM 305 13.4 Sequential Minimal Optimization Algorithm 308 13.5 Multi-Class Classification 316 13.6 Kernelized Bayes classifier 319 13.7 Hierarchical (Tree) Classifiers 325

14  Clustering Analysis  329

14.1 K-means clustering 329 14.2 Gaussian mixture model 333 14.3 Mixture of Bernoulli 341 14.4 General EM Algorithm 344 14.5 Linear Models for Binary Classification 345

Part V Neural Networks 349

15  Biologically Inspired Networks  351

15.1 Biological Inspiration 351 15.2 Hebbian Learning 354 15.3 Hopfield Network 356

16  Perceptron-Based Networks  360

16.1 Perceptron 360 16.2 Back Propagation 367 16.3 Autoencoder 375 16.4 Deep Learning 381 16.5 Generative Adversarial Networks 384

17  Competition-Based Networks  386

17.1 Competitive Learning Network 386 17.2 Self-Organizing Map (SOM) 390

Part VI Reinforcement Learning 395

18  Introduction to Reinforcement Learning  397

18.1 Reinforcement Learning 397 18.2 Markov Decision Process 398 18.3 Model-Based Planning 404 18.4 Model-Free Evaluation and Control 408

18.4.1 Monte Carlo (MC) Algorithms 412 18.4.2 Temporal Difference (TD) Algorithms 415 18.4.3 TD(λ) Algorithm 418

18.5 Value Function Approximation 423 18.6 Control based on Function Approximation 427 18.7 Deep Q-learning 429 18.8 Policy Gradient Methods 429

Appendix A A Review of Linear Algebra 437 A.1 Inner Product Space 437 A.2 Orthogonal Basis and Gram-Schmidt Process 444 A.3 Rank, Trace, Determinant, Transpose, and Inverse of Matrices 446 A.4 Normal, Unitary, and Similar Matrices 448 A.5 Positive/Negative (Semi)-Definite Matrices 449 A.6 Woodbury Matrix Identity and Sherman-Morrison Formula 450 A.7 Inverse and Determinant of Partitioned Symmetric Matrices 451 A.8 Unitary Transform 453 A.9 Eigenvalue Decomposition 454 A.10 Generalized Eigenvalue Problem 459 A.11 Rayleigh Quotient 461 A.12 Normal Matrices and Diagonalizability 461 A.13 Centering Matrix 463 A.14 Singular Value Decomposition 464 A.15 Pseudo-Inverse 468 A.16 Vector Norms 469 A.17 Matrix Norms 473 A.18 Vector and Matrix Differentiation 481 A.19 The Fundamental Theorem of Linear Algebra 482 A.20 Pseudo-Inverse Solutions Based on SVD 494 A.21 Rank-Nullity Theorem 501 A.22 Taylor Series Expansion 503 A.23 Normal Direction of a Plane 505 A.24 Convexity 506 A.25 Jensen Inequality 508 A.26 Sensitivity and Conditioning 509 A.27 Some Useful Inequalities 517

Notes 520 Author index 521 Subject index 522

Illustrations

1.1  Bisection Search  6
1.2  7

Tables Boxes List of contributors Notation

AFM atomic force microscope
AKPZ anisotropic KPZ equation
a_{0}  lattice constant
c_{q}(ℓ) q-th order correlation function
d_{E}  embedding dimension
d_{f}  fractal dimension
L  system size
≡  defined to be equal
∼  asymptotically equal (in scaling sense)
≈  approximately equal (in numberical value)

Part I Mathematical Fundations 1 Solving Equations

In this chapter we consider the methods for solving a set of M simultaneous equations of N variables:

\{ \begin{matrix}f_{1}(x_{!}, ) = f_{1}(x) = 0 \\ ⋯ & ⋯ & ⋯ \\ f_{M}(x_{1}, ⋯, x_{N})) = f_{M}(x) = 0 \end{matrix}  (1.1)

wherex = [x_{1}, ⋯, x_{N}]^{T} ∈ ℝ^{N} is an N-D column vector, a point in the N-D vector space ℝ^{N} spanned by {x_{1}, ⋯, x_{N}}. We further define f = [f_{1}, ⋯, f_{M}]^{T} as an M-D column vector, and represent this equation system more concisely as f(x) = 0.

Geometrically, a function f(x) defined over this space ℝ^{N} is a hypersurface in the N+1 dimensional space, of which the N+1st dimension represents the function value y = f(x) corresponding to a point x in the N-D space. The roots or zeros of function f(x), i.e., the solutions of equation f(x) = 0, are all the points on a hypercurve in the N-D space, as the intersection of the hypersurface y = f(x) and the hyperplane y = 0, if they do intersect.

In general, no closed-form solution exists for a nonlinear equation system and the number of solutions is unknown. We therefore have to rely on numerical methods to find iteratively one or more such solutions if they do exist from an initial guess of the solution.

For example, in the special case of M = N = 2, functions f_{1}(x_{1}, x_{2}) and f_{2}(x_{1}, x_{2}) are two surfaces defined over the 2-D space spanned by x_{1} and x_{2}, and the roots of each of the two functions are on a curve as the intersection of the corresponding surface and the 2-D space. The solutions of the system composed of the two equations f_{1} = 0 and f_{2} = 0 are the intersection of these two curves, if they do intersect, otherwise no solution exists.

Example: Consider a simultaneous equation system:

\{ \begin{matrix}f_{1}(x, y) = x^{2} + y^{2} − 2 = 0 \\ f_{2}(x, y) = x + y − C = 0 \end{matrix}

The first function f_{1}(x, y) is a parabolic cone centrally symmetric to the vertical axis, and its roots form a circle x^{2} + y^{2} = 2 on the x-y plane centered at the origin (0, 0) with radius 1; the second function f_{2}(x, y) is a plane through the origin, and its roots form a straight line y = C − x on the x-y plane. The solutions of the equation system are where the two curves intersect:

• If C = 0, there are two solutions (1, −1) and (−1, 1);

• If C = 2, there is only one solution (1, 1);

• If C = 3, the two curves do not intersect, i.e., no solution exists.

1.1 Linear Equation Systems

Consider a linear equation system of M equations and N variables:

\{ \begin{matrix}∑_{j=1}^{N} a_{1j}x_{j} − b_{1} = 0 \\ ⋯ \\ ∑_{j=1}^{N} a_{Mj}x_{j} − b_{M} = 0 \end{matrix}  (1.2)

which can be expressed in vector form:

f(x) = Ax − b = 0  (1.3)

where

A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ] ,  x = [ x_{1} ; ⋮ ; x_{N} ] ,  b = [ b_{1} ; ⋮ ; b_{M} ]  (1.4)

In general, the existence and uniqueness of the solution x can be determined by the fundamental theorem of linear algebra, (see Table 3 in Section A.19), based on the rank R of the coefficient matrix A and whether the system is underdetermined (underconstrained) if M < N, or overdetermined (overcostrained) if M > N.

Specially if R = M = N, A is full rank square matrix and its inverse A^{−1} exists. Then the system has a unique solution x = A^{−1}b.

But if M > N = R and b is not in the column space of A, the system is overconstrained without a solution. In this case we can still find an optimal approximated solution. We first define the residual of the system as r = b − Ax, and then define the sum-of-squares error (SSE) as:

ε(x) = \frac{1}{2}||r||^{2} = \frac{1}{2}r^{T} r = \frac{1}{2}(b − Ax)^{T} (b − Ax)
= \frac{1}{2} (b^{T} b − x^{T} A^{T} b − b^{T} Ax + x^{T} A^{T} Ax)  (1.5)

To find the optimal solution x^{∗} that minimizes ε, we set its derivative with respect to x to zero (see section A.18 for differentiation with respect to a vector variable):

\frac{d}{dx}ε(x) = \frac{1}{2} \frac{d}{dx}||r||^{2} = \frac{1}{2} \frac{d}{dx} (b^{T} b − x^{T} A^{T} b − b^{T} Ax + x^{T} A^{T} Ax)
= −A^{T} b + A^{T} Ax = 0  (1.6)

Solving this matrix equation we get

x = (A^{T} A)^{−1} A^{T} b = A^{−}b  (1.7)

where A^{−} = (A^{T} A)^{−1} A^{T} is the pseudo-inverse (Section A.15) of the nonsquare matrix A.

If the M equations are barely independent, is the N × N matrix A^{T} A it still has a full rank R = N and is invertible, but it is a near-singular matrix. In this case some of its eigenvalues is very close to zero, and correspondingly some of the eigenvalues of its inverse (A^{T} A)^{−1} may be huge. Consequently the system becomes ill-conditioned or ill-posed, in the sense that a small change in the system (in either A or b) due to noise may cause a large change in the solution x.

This ill-posted problem can be addressed by regularization, by which the solution x is under control so that it will not take unreasonably high values. Specifically we construct an objective function that contains a penalty term for large x, as well as the error term ε = ||r||^{2}/2:

J(x) = \frac{1}{2}||r||^{2} + \frac{λ}{2}||x||^{2}  (1.8)

By minimizing J(x), we can obtain a solution x of small norm ||x|| as well as low error ε(x). Same as before, the solution x can be obtained by setting the derivative of J(x) to zero

\frac{d}{dx}J(x) = −A^{T} b + A^{T} Ax + λx = 0  (1.9)

and solving the resulting equation to get:

x = (A^{T} A + λI)^{−1}A^{T} b  (1.10)

We see that even if A^{T} A is near singular, matrix A^{T} A + λI is not due to the additional term λI.

By adjusting the hyperparameter λ, we can make a proper tradeoff between accuracy and stability.

• Small λ: the solution is more accurate but less stable as it is more prone to noise, i.e., the variance error may be large. This is called overfitting;

• Large λ: the solution is less accurate but more stable as it is less affacted by noise. This is called underfitting.

The issue of overfitting versus underfitting is of central importance in machine learning in general, and will be more formally addressed while discussing varioius regression and classification algorithms in some later Chapters.

Figure 1.1 Bisection Search

1.2 The Bisection and Secant methods

Here we consider a set of methods that find the solution x^{∗} of a single-variable nonlinear equation f(x) = 0, by searching iteratively through a neighborhood of the domain, in which x^{∗} is known to be located.

The bisection search

This method requires two initial guesses x_{0} < x_{1} satisfying f(x_{0})f(x_{1}) < 0. As f(x_{0}) and f(x_{1}) are on opposite sides of the x-axis y = 0, the solution x^{∗} at which f(x^{∗}) = 0 must reside somewhere in between of these two guesses, i.e., x_{0} < x^{∗} < x_{1}.

Given such two end points x_{0} and x_{1}, we find another point x_{2} in between and use it to replace one of the end points at which the function value f(x) has the same sign as that of f(x_{2}). The new search interval is a = x_{2} − x_{0} if f(x_{2})f(x_{1}) > 0, or b = x_{1} − x_{2} if f(x_{0})f(x_{2}) > 0, either of which is smaller than the previous interval a + b = x_{1} − x_{0}. This process is then carried out iteratively until the solution is eventually approached, with a guaranteed convergence.

While any point between the two end points can be chosen for the next iteration, we want to avoid the worst possible case in which the solution always happens to be in the larger of the two sections a and b. We can therefore choose the middle point between the two end points, i.e., x_{2} = (x_{0} + x_{1})/2, so that a = b = (x_{1} − x_{0})/2, i.e., the new search interval is always halved in each step of the iteration:

x_{n+1} = \frac{x_{n} + x_{n−1}}{2}  (1.11)

Here, without loss of generality, we have assumed f(x_{n−1})f(x_{n+1}) > 0 and x_{n−1} is replaced by x_{n+1}.

For example, we assume the root is located at x^{∗} = 2.2 between the two initial values are x_{0} = 0 and x_{1} = 8, then we get

Figure 1.2

\begin{matrix}n & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 \\ x_{n} & 0 & 8 & 4 & 2 & 3 & 2.5 & 2.25 & 2.125 & ⋯ \\ |e_{n}| & 2.2 & 5.8 & 1.8 & 0.2 & 0.8 & 0.3 & 0.05 & 0.075 & ⋯ \end{matrix}  (1.12)

Note that the error |e_{n}| = |x_{n} − x^{∗}| does not necessarily always reduce monotonically. However, as in each iteration the search interval is always halved, the worst possible error is also halved:

|e_{n+1}| = |x_{n+1} − x^{∗}| ≤ \frac{|x_{n} − x_{n−1}|}{2} = \frac{|x_{0} − x_{1}|}{2^{n}} ≈ \frac{|e_{n}|}{2}  (1.13)

i.e., the convergence of the iteration is linear (of order p = 1) and the rate of convergence is 1/2:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{p}} = lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|} ≤ μ = \frac{1}{2}  (1.14)

Compared to other methods to be considered later, the bisection method converges rather slowly, but one of the advantages of the bisection method is that no derivative of the given function is needed. This means the given function f(x) does not need to be differentiable.

The Secant method

Same as in the bisection method, here again we assume there are two initial values x_{0} and x_{1} available, but they do not have to satisfy f(x_{0})f(x_{1}) < 0. Here the iteration is based on the zero-crossing of the secant line passing through the two points f(x_{0}) and f(x_{1}), instead of their middle point. The equation for the secant is:

\frac{y − f(x_{0})}{x − x_{0}} = \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}}  (1.15)

At the zero crossing, y = 0 and the equation above can be solved for x, the position of the zero-crossing:

x = x_{0} − \frac{x_{1} − x_{0}}{f(x_{1}) − f(x_{0})}f(x_{0}) = x_{0} − \frac{f(x_{0})}{f̂′(x_{0})}  (1.16)

where f̂′(x_{0}) is the finite difference that approximates the derivative f′(x_{0}):

f̂′(x_{0}) = \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}} = \frac{Δf(x_{0})}{Δx_{0}} ⟹^{Δx_{0}→0} f′(x_{0})  (1.17)

The new value x obtained above, as a better estimate of the root than either x_{0} or x_{1}, is used to replace one of them:

• If f(x_{0})f(x_{1}) < 0, then x_{i} (i = 0, 1) satisfying f(x_{i})f(x) > 0 is replaced by x_{2} = x (same as the bisection method);

• If f(x_{0})f(x_{1}) > 0, then x_{i} (i = 0, 1) with greater |f(x_{i})| is replaced by x_{2} = x.

This process is then carried out iteratively to approach the root:

x_{n+1} = x_{n} − \frac{f(x_{n})}{f̂′(x_{n})}  (1.18)

In case f(x_{n}) = f(x_{n−1}), the approximated derivative is zero f̂′(x_{n}) = 0, and x_{n+1} cannot be found. In such a case, a root may exist between x_{n} and x_{n−1}. Therefore the way to resolve this problem is to combine the bisection search with the secant method so that when f̂′ = 0, the algorithm will switch to bisection search. This is Dekker’s method:

x_{n+1} = \{ \begin{matrix}x_{n} − f(x_{n})/f̂′(x_{n}) & if f(x_{n}) ≠ f(x_{n−1}) \\ (x_{n} + x_{n−1})/2 & otherwise \end{matrix}  (1.19)

We now consider the order of convergence of the secant method. Let x^{∗} be the root at which f(x^{∗}) = 0. The error of x_{n+1} is:

e_{n+1} = x_{n+1} − x^{∗} = x_{n} − \frac{x_{n} − x_{n−1}}{f(x_{n}) − f(x_{n−1})} f(x_{n}) − x^{∗}
= \frac{(x_{n−1} − x^{∗})f(x_{n}) − (x_{n} − x^{∗})f(x_{n−1})}{f(x_{n}) − f(x_{n−1})} = \frac{e_{n−1}f(x_{n}) − e_{n}f(x_{n−1})}{f(x_{n}) − f(x_{n−1})} (1.20)

Consider the Taylor expansion of f(x_{n}):

f(x_{n}) = f(x^{∗}+e_{n}) = f(x^{∗})+f′(x^{∗})e_{n}+\frac{1}{2}f′′(x^{∗})e_{n}^{2}+O(e_{n}^{3}) = f′(x^{∗})e_{n}+\frac{1}{2}f′′(x^{∗})e_{n}^{2}+O(e_{n}^{3})

(1.21)

(as f(x^{∗}) = 0) and similarly

f(x_{n−1}) = f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})  (1.22)

Substituting these into the expression for e_{n+1} above we get

e_{n+1} = \frac{e_{n−1}[f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})] − e_{n}[f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})]}{[f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})] − [f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})]}
= \frac{e_{n−1}e_{n}f′′(x^{∗})(e_{n} − e_{n−1})/2 + O(e_{n−1}^{4})}{(e_{n} − e_{n−1})f′(x^{∗}) + (e_{n}^{2} − e_{n+1}^{2})f′′(x^{∗})/2 + O(e_{n−1}^{3})}
= \frac{e_{n−1}e_{n}f′′(x^{∗})/2 + O(e_{n−1}^{3})}{f′(x^{∗}) + (e_{n} + e_{n+1})f′′(x^{∗})/2 + O(e_{n−1}^{2})}
= \frac{e_{n−1}e_{n}f′′(x^{∗})/2 + O(e_{n−1}^{3})}{f′(x^{∗}) + O(e_{n}) + O(e_{n−1}^{2})}  (1.23)

When n → ∞, the lowest order terms in both the numerator and denominator become the dominant terms as all other higher order terms approach to zero, and we have

e_{n+1} = \frac{e_{n−1}e_{n}f′′(x^{∗})}{2f′(x^{∗})} = C e_{n}e_{n−1}  (1.24)

where we have defined a constant C = f′′(x^{∗})/2f′(x^{∗}). To find the order of convergence, we need to find p in

|e_{n+1}| ≤ |C| |e_{n−1}| |e_{n}| = μ|e_{n}|^{p}  (1.25)

Solving this equation for |e_{n}| we get

|e_{n}| = (\frac{|C|}{μ}|e_{n−1}|)^{1/(p−1)} = (\frac{|C|}{μ})^{1/(p−1)} |e_{n−1}|^{1/(p−1)}  (1.26)

On the other hand, when n → ∞ we also have

|e_{n}| = μ|e_{n−1}|^{p}  (1.27)

Equating the right-hand sides of the two equations above we get

(\frac{|C|}{μ})^{1/(p−1)} |e_{n−1}|^{1/(p−1)} = μ|e_{n−1}|^{p}  (1.28)

which requires the following two equations to hold:

p = \frac{1}{p − 1},  μ = (\frac{|C|}{μ})^{1/(p−1)} = (\frac{|C|}{μ})^{p}  (1.29)

These two equations can be solved separately to get

p = \frac{1 + \sqrt{5}}{2} = 1.618, μ = |C|^{p/(p+1)} = C^{(\sqrt{5}−1)/2} = |C|^{0.618}  (1.30)

i.e.,

|e_{n+1}| = μ|e_{n}|^{p} = |C|^{0.618}|e_{n}|^{1.618} = |\frac{f′′(x)}{2f′(x)}|^{0.618}|e_{n}|^{1.618}  (1.31)

or

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{1.618}} = μ = |C|^{0.618}  (1.32)

We see that the rate of convergence for the secant method is μ = |C|^{0.618} and the order of convergence is p = 1.618, which is better than the linear convergence of the bisection search (which does not use the information of the specific function f(x)), but worse than quadratic convergence of the Newton-Raphson method based on the function derivative f′(x) instead of the approximation f̂′(x), to be considered in the next section.

The inverse quadratic interpolation

Similar to the secant method that approximates the given function f(x) by a straight line that goes through two consecutive points {x_{n}, f(x_{n})} and {x_{n−1}, f(x_{n−1})}, the inverse quadratic interpolation method approximates the function by a quadratic curve that goes through three consecutive points {x_{i}, f(x_{i})), (i = n, n−1, n− 2)}. As the function may be better approximated by a quadratic curve rather than a straight line, the iteration is likely to converge more quickly.

In general, any function y = f(x) can be approximated by a quadratic function q(x) based on three points at y_{0} = f(x_{0}), y_{1} = f(x_{1}), and y_{2} = f(x_{2}) by the Lagrange interpolation:

y = q(x) = \frac{(x − x_{1})(x − x_{2})}{(x_{0} − x_{1})(x_{0} − x_{2})}y_{0} + \frac{(x − x_{0})(x − x_{2})}{(x_{1} − x_{0})(x_{1} − x_{2})}y_{1} + \frac{(x − y_{0})(x − x_{1})}{(x_{2} − x_{0})(x_{2} − x_{1})}y_{2}

(1.33)

However, here we use the interpolation x = q^{−1}(y) to approximate the inverse (horizontal) function x = f^{−1}(y):

x = q^{−1}(y) = \frac{(y − y_{1})(y − y_{2})}{(y_{0} − y_{1})(y_{0} − y_{2})}x_{0}+ \frac{(y − y_{0})(y − y_{2})}{(y_{1} − y_{0})(y_{1} − y_{2})}x_{1}+ \frac{(y − y_{0})(y − y_{1})}{(y_{2} − y_{0})(y_{2} − y_{1})}x_{2}

(1.34)

Obviously when y = 0, the corresponding x is the zero-crossing of x = q^{−1}(y):

x_{3} = q^{−1}(y)|_{y=0} = \frac{y_{1}y_{2}}{(y_{0} − y_{1})(y_{0} − y_{2})}x_{0}+\frac{y_{0}y_{2}}{(y_{1} − y_{0})(y_{1} − y_{2})}x_{1}+\frac{y_{0}y_{1}}{(y_{2} − y_{0})(y_{2} − y_{1})}x_{2}

(1.35)

which can be used as an estimate of the root x^{∗} of y = f(x). This expression can then be converted into an iteration by which the next root estimate x_{n+1} is computed based on the previous three estimates at x_{n}, x_{n−1}, and x_{n−2}.

1.3 Fixed point iteration

We first consider the special case of solving the equation f(x) = 0 when M = N = 1. To find a solution x^{∗} that satisfies the quation f(x) = 0, we can first convert it into an equivalent equation g(x) = x, in the sense that an x satisfying one of the equations will also satisfy the other, and then carry out an iteration x_{n+1} = g(x_{n}) from some initial value x_{0}. If the iteration converges at a point x^{∗}, i.e., g(x^{∗}) = x^{∗}, then we also have f(x^{∗}) = 0, i.e., x^{∗} is a solution of the equation f(x) = 0. Consider the following examples:

Example 1

To solve the equation

f(x) = log(x) − 0.5 = 0

we can first construct another function

g(x) = x − f(x) = x − log(x) + 0.5

so that the equation g(x) = x is indeed equivalent to the given equation f(x) = 0. We then carry out the iteration x_{n+1} = g(x_{n}) from some initial value, such as x_{0} = 0.5, and get:

\begin{matrix}n & x_{n} & g(x_{n}) & f(x_{n}) \\ 0 & 0.50000 & 1.69315 & −1.19315 \\ 1 & 1.69315 & 1.66656 & 0.02659 \\ 2 & 1.66656 & 1.65580 & 0.01076 \\ 3 & 1.65580 & 1.65152 & 0.00428 \\ 4 & 1.65152 & 1.64982 & 0.00169 \\ 5 & 1.64982 & 1.64915 & 0.00067 \\ 6 & 1.64916 & 1.64889 & 0.00026 \\ 7 & 1.64889 & 1.64879 & 0.00010 \\ 8 & 1.64879 & 1.64875 & 0.00004 \\ 9 & 1.64875 & 1.64873 & 0.00002 \\ 10 & 1.64873 & 1.64873 & 0.00001 \\ 11 & 1.64873 & 1.64872 & 0.00000 \\ 12 & 1.64872 & 1.64872 & 0.00000 \end{matrix}

We see that the iteration converges to x^{∗} = lim_{n→∞} g(x_{n}) = 1.64872 satisfying g(x^{∗}) = x^{∗} − f(x^{∗}) = x^{∗}, and, equivalently, x^{∗} is also the solution of the given equation f(x) = 0.5 − log(x) = 0:

0.5 − log(1.64872) = 0, i.e., e^{0.5} = \sqrt{e} = 1.64872

Alternatively, we could construct a different function also equivalent to f(x) = 0:

g(x) = x + f(x) = x + log(x) − 0.5

But this iteration no longer converges. Why does it not work?

Example 2

f(x) = e^{x} − 1/x = 0

We first convert this equation into an equivalent form

g(x) = x = e^{−x}

and then carry out the iteration:

x_{n+1} = g(x_{n}) = e^{−x_{n}} ⟹^{n→∞} x^{∗} = 0.5671

which is the root of the given equation f(x) = e^{x} − 1/x = 0, i.e., e^{−0.5671} = 0.5671.

Alternatively, the given equation can also be converted into a different form g(x) = x = − ln(x). However, the iteration based on this function no longer converges.

Why does this iterative method work in some cases but fails in others? To answer these questions we need to understand the theory behind the method, the fixed point of a contraction function.

If a single variable function g(x) satisfies

|g(x_{1}) − g(x_{2})| ≤ k|x_{1} − x_{2}|, k ≥ 0  (1.36)

it is Lipschitz continuous, and k is a Lipschitz constant. Specially, if k = 1, then g(x) is a non-expansive function, if 0 ≤ k < 1, then g(x) is a contraction function or simply a contraction. These concepts can be generated to functions of multivariables x = [x_{1}, ⋯, x_{N}]^{T} ∈ ℝ^{N} (a point in an N-D metric space V):

g_{n}(x) = g_{n}(x_{1}, ⋯, x_{N}),  (n = 1, ⋯, N)  (1.37)

which can also be expressed in vector form:

g(x) = [g_{1}(x), ⋯, g_{N}(x)]^{T}  (1.38)

Definition: In a metric space V with certain distance d(x, y) defined between any two points x, y ∈ V, a function g : V → V is a contraction if

d(g(x), g(y)) ≤ k d(x, y)  (1.39)

The smallest k value that satisfies the above is called the (best) Lipschitz constant.

Intuitively, a contraction reduces the distance between points in the space, i.e., it brings them closer together. A function g(x) may not be a contraction through out its entire domain, but it can be a contraction in the neighborhood of a certain point x^{∗} ∈ V, in which any x is sufficiently close to x^{∗} so that

d(g(x), g(x^{∗})) ≤ k d(x, x^{∗}), 0 ≤ k < 1, when d(x, x^{∗}) < ϵ > 0 (1.40)

Definition: A fixed point x^{∗} of a function g(x) is a point in its domain that is mapped to itself:

x^{∗} = g(x^{∗})  (1.41)

We immediately have

g(g(x^{∗})) = g^{2}(x^{∗}) = x^{∗}, · · · · · · , g(g(g(⋯ g(x^{∗}) ⋯))) = g^{n}(x^{∗}) = x^{∗} (1.42)

A fixed point x^{∗} is an attractive fixed point if any point x in its neighborhood converges to x^{∗}, i.e., lim_{n→∞} g^{n}(x) = x^{∗}.

Fixed Point Theorem : Let g(x) be a contraction function satisfying

d(g(x) − g(y)) ≤ k d(x − y) (0 ≤ k < 1)  (1.43)

then there exists a unique fixed point x^{∗} = g(x^{∗}), which can be found by an iteration from an arbitrary initial point x_{0}:

lim_{n→∞} x_{n} = lim_{n→∞} g(x_{n−1}) = lim_{n→∞} g^{2}(x_{n−2}) = ⋯ = lim_{n→∞} g^{n}(x_{0}) = x^{∗} (1.44)

Proof

Here the distance d(x − y) is specifically defined as the p-norm (section A.16) of the vector x − y:

d(x, y) = ||x − y||_{p} = (∑_{n=1}^{N}(x_{n} − y_{n})^{p})^{1/p}  (1.45)

where p ≥ 1, e.g., p = 1, 2, ∞. Also, for conveninece, we can drop p so that d(x, y) = ||x − y||.

• We first prove constructively the existence of a fixed point. As g(x) is a contraction, we have

||x_{n+1}−x_{n}|| = ||g(x_{n})−g(x_{n−1})|| ≤ k||x_{n}−x_{n−1}|| ≤ k^{2}||x_{n−1}−x_{n−2}|| ≤ ⋯ ≤ k^{n}||x_{1}−x_{0}||

(1.46)

As 0 ≤ k < 1, we have lim_{n→∞} ||x_{n+1} − x_{n}|| = 0. This is a Cauchy sequence (Section A.1) that converges to some point lim_{n→∞} x_{n} = x^{∗} also in the space. We further have

g(x^{∗}) = g( lim_{n→∞} x_{n}) = lim_{n→∞} g(x_{n}) = lim_{n→∞} x_{n+1} = x^{∗}  (1.47)

i.e., the limit of the Cauchy sequence lim_{n→∞} x_{n} = x^{∗} is a fixed point.

• We next prove the uniqueness of the fixed point. Let x_{1}^{∗} and x_{2}^{∗} be two fixed points of g(x), then we have

||g(x_{1}^{∗}) − g(x_{2}^{∗})|| ≤ k||x_{1}^{∗} − x_{2}^{∗}|| = k||g(x_{1}^{∗}) − g(x_{2}^{∗})||  (1.48)

For any k ≠ 0, the above holds only if ||x_{1}^{∗} − x_{2}^{∗}|| = ||g(x_{1}^{∗}) − f(x_{2}^{∗})|| = 0,

i.e., x_{1}^{∗} = x_{2}^{∗} is the unique fixed point.

QED

Theorem: Let x^{∗} = g(x^{∗}) be a fixed point of a differentiable function g(x), i.e, ∂g_{i}/∂x_{j} exists for any 1 ≤ i, j ≤ N. If the norm of the Jacobian matrix is smaller than 1, ||J_{g}(x^{∗})|| < 1, then g(x) is a contraction at x^{∗}.

The Jacobian matrix of g(x) is defined as

g′(x) = J_{g}(x) = [ \begin{matrix}\frac{∂g_{1}}{∂x_{1}} & ⋯ & \frac{∂g_{1}}{∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂g_{N}}{∂x_{1}} & ⋯ & \frac{∂g_{N}}{∂x_{N}} \end{matrix} ]  (1.49)

Proof:

Consider the Taylor expansion (Section A.22) of the function g(x) in the neighborhood of x^{∗}:

g(x) = g(x^{∗})+g′(x^{∗})(x−x^{∗})+R(x−x^{∗}) = g(x^{∗})+J_{g}(x^{∗})(x−x^{∗})+R(x−x^{∗})

(1.50)

where R(x − x^{∗}) is the remainder composed of second and higher order terms of δ = x − x^{∗}. Subtracting g(x^{∗}) and taking any p-norm on both sides, we get

||g(x) − g(x^{∗})||_{p} = ||J_{g}(x^{∗})(x − x^{∗}) + R(x − x^{∗})||_{p}  (1.51)

When x → x^{∗}, the second and higher order terms of x − x^{∗} disappear and R(x − x^{∗}) → 0, we have

||g(x) − g(x^{∗})||_{p} = ||J_{g}(x^{∗})(x − x^{∗})||_{p} ≤ ||J_{g}(x^{∗})||_{p} · ||x − x^{∗}||_{p},  (1.52)

The inequality is due to the Cauchy-Schwarz inequality (Section A.1) if ||J_{g}(x^{∗})|| < 1, the function g(x) is a contraction at x^{∗}.

QED

In particular, for a single-variable function in an N = 1 dimensional space, we have

g(x) − g(x^{∗}) = g′(x^{∗})(x − x^{∗}) + \frac{1}{2}g′′(x^{∗})(x − x^{∗})^{2} + R(x − x^{∗})  (1.53)

and

|g(x) − g(x^{∗})| ≤ |g′(x^{∗})(x − x^{∗})| ≤ |g′(x^{∗})| |(x − x^{∗})|  (1.54)

If |g′(x)| < 1, then g(x) is a contraction at x^{∗}.

Now we understand why in the examples of the previous section the iteration leads to convergence in some cases but divergence in other cases: if |g′(x)| < 1, the iteration will converge to the root x^{∗} of f(x) = 0, but if |g′(x)| > 1, it never will never converge.

The iterative process x_{n+1} = g(x_{n}) for finding the fixed point of a singlevariable function g(x) can be shown graphically as the intersections of the function y = g(x) and the identity function y = x, as shown below. The iteration converges in the first two cases as |g′(x)| < 1, but it diverges in the last two cases as |g′(x)| > 1.

We next find the order of convergence of the fixed point iteration. Consider

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|} = lim_{n→∞} \frac{|x_{n+1} − x^{∗}|}{|x_{n} − x^{∗}|} = lim_{n→∞} \frac{|g(x_{n}) − g(x^{∗})|}{|x_{n} − x^{∗}|} = |g′(x^{∗})| = μ < 1

(1.55)

We see that in general the fixed point iteration converges linearly. However, if the iteration function g(x) has zero derivative at the fixed point g′(x^{∗}) = 0, we have

g(x)−g(x^{∗}) = g′(x^{∗})(x−x^{∗})+\frac{1}{2}g′′(x^{∗})(x−x^{∗})^{2}+R(x−x^{∗}) = \frac{1}{2}g′′(x^{∗})(x−x^{∗})^{2}+R(x−x^{∗})

(1.56)

and the iteration converges quadratically:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} = lim_{n→∞} \frac{|x_{n+1} − x^{∗}|}{|x_{n} − x^{∗}|^{2}} = lim_{n→∞} \frac{|g(x_{n}) − g(x^{∗})|}{|x_{n} − x^{∗}|^{2}} = \frac{1}{2}|g′′(x^{∗})| = constant

(1.57)

Moreover, if g′′(x^{∗}) = 0, then the iteration converges cubically.

Consider a specific iteration function in the form of g(x) = x−f(x)ϕ(x), which is equivalent to f(x) = 0, as if x^{∗} is the root of f(x) satisfying f(x^{∗}) = 0, it also satisfies g(x^{∗}) = x^{∗}, which is indeed a fixed point of g(x). The derivative of this function is

g′(x) = (x − f(x)ϕ(x))′ = 1 − f′(x)ϕ(x) − f(x)ϕ′(x) = 1 − f′(x)ϕ(x) (1.58)

If f′(x) ≠ 0, we can define ϕ(x) = 1/f′(x) so that g′(x) = 1 − f′(x)/f′(x) = 0, then the convergence becomes quadratic. This is actually the Newton-Raphson method discussed in the following section.

Example 1

Find the solution of the following equation:

f(x) = x^{3} − x − 2 = 0

This equation can converted into an equivalent form of g(x) = x in two different ways:

g_{1}(x) = x = \sqrt[3]{x + 2}, and g_{2}(x) = x = x^{3} − 2

In the figure below, the two functions g_{1}(x) (left) and g_{2}(x) (right) together with f(x) and the identity function are plotted:

The iteration based on g_{1}(x) converges to the solution x^{∗} = 1.5214 for any initial guess −∞ < x_{0} < ∞, as g_{1}(x) is a contraction. However, the iteration based on g_{2}(x) does not converge to x^{∗} as it is not a contraction in the neighborhood of x^{∗}. In fact, the iteration will diverge towards either −∞ if x_{0} < x^{∗} or ∞ if x_{0} > x^{∗}.

Example 2

To solve the following equation

f(x) = x^{3} − 3x + 1 = 0

we first convert it into the form of g(x) = x in two different ways:

g_{1}(x) = x = \sqrt[3]{3x − 1}, g_{2}(x) = x = \frac{1}{3}(x^{3} − 1)

As can be seen in the plots, this equation has three solutions,

x_{1} = −1.8794, x_{2} = 0.3473, x_{3} = 1.5321

of which x_{1} and x_{3} can be obtained by the iteration based on g_{1}(x) and x_{2} can be obtained by the iteration based on g_{2}(x). But neither of them can find all three roots.

• As shown in the plot on the left, |g′_{1}(x)| < 1 for all −∞ < x < ∞ except in the neighborhood of x_{2} = 0.3473, i.e., g_{1}(x) is a contraction mapping everywhere except around x_{2}. Therefore the iteration starting from any initial guess x_{0} will converge to either x_{1} = −1.8749 if x_{0} < x_{2}, or x_{3} = 1.5321 if x_{0} > x_{2}.

x_{n+1} = g_{1}(x_{n}) ⟹^{n→∞} \{ \begin{matrix}x_{1} = −1.8749 & x_{0} < x_{2} \\ x_{3} = 1.5321 & x_{0} > x_{2} \end{matrix}

However, as g_{1}(x) is not a contraction mapping around x = x_{2}, the iteration will never converge to x_{2}.

• As shown in the plot on the right, |g′_{2}(x)| > 1 for all −∞ < x < ∞ except in the neighborhood of x_{2}, i.e., g_{2}(x) is not a contraction mapping around either x_{1} or x_{3}. Therefore the iteration based on g_{2}(x) will not converge to either x_{1} or x_{3}, but it may converge to x_{2}, if the initial guess x_{0} is in the range x_{1} < x_{0} < x_{3}. However, if x_{0} is outside this range the iteration will diverge toward either −∞ if x_{0} < x_{1} of ∞ if x_{0} > x_{3}.

x_{n+1} = g_{2}(x_{n}) ⟹^{n→∞} \{ \begin{matrix}−∞ & x_{0} < x_{1} \\ x_{2} = 0.3473 & x_{1} < x_{0} < x_{3} \\ ∞ & x_{0} > x_{3} \end{matrix}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 1 & −2.0408275e + 00 & −1.3775173e + 00 \\ 2 & −1.9240239e + 00 & −3.5041089e − 01 \\ 3 & −1.8919392e + 00 & −9.6254050e − 02 \\ 4 & −1.8829328e + 00 & −2.7019276e − 02 \\ 5 & −1.8803890e + 00 & −7.6311660e − 03 \\ 6 & −1.8796694e + 00 & −2.1590449e − 03 \\ 7 & −1.8794656e + 00 & −6.1114717e − 04 \\ 8 & −1.8794080e + 00 & −1.7301761e − 04 \\ 9 & −1.8793916e + 00 & −4.8983741e − 05 \\ 10 & −1.8793870e + 00 & −1.3868146e − 05 \\ 11 & −1.8793857e + 00 & −3.9263250e − 06 \\ 12 & −1.8793853e + 00 & −1.1116151e − 06 \end{matrix}

Example 3

f(x) = sin(x) − x^{3} = 0

This equation can be converted into the form g(x) = x in different ways:

•

g_{0}(x) = x = \sqrt[3]{sin(x)}, g′_{0}(x) = \frac{cos(x)}{3 sin(x)^{2/3}}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 2.000000e + 00 & −7.0907 \\ 1 & 9.688027e − 01 & −0.085089 \\ 2 & 9.375886e − 01 & −0.018075 \\ 3 & 9.306842e − 01 & −0.0041048 \\ 4 & 9.291018e − 01 & −0.00094611 \\ 5 & 9.287364e − 01 & −0.00021881 \\ 6 & 9.286518e − 01 & −5.0647e − 05 \\ 7 & 9.286322e − 01 & −1.1725e − 05 \\ 8 & 9.286277e − 01 & −2.7144e − 06 \end{matrix}

•

g_{1}(x) = x = sin(x)/x^{2}, g′_{1}(x) = \frac{cos(x)}{x^{2}} − \frac{2 sin(x)}{x^{3}}

•

g_{2}(x) = x + sin(x) − x^{3}, g′_{2}(x) = cos(x) − 3 x^{2} + 1

•

g_{3}(x) = x − \frac{sin(x) − x^{3}}{cos(x) − 3x^{2}} , g′_{3}(x) = −\frac{(6x + sin(x))(sin(x) − x^{3})}{(cos(x) − 3x^{2})^{2}}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 2.000000e + 00 & −7.0907 \\ 1 & 1.428913e + 00 & −1.9276 \\ 2 & 1.106787e + 00 & −0.46152 \\ 3 & 9.637850e − 01 & −0.073886 \\ 4 & 9.304466e − 01 & −0.0036294 \\ 5 & 9.286316e − 01 & −1.0509e − 05 \\ 6 & 9.286263e − 01 & −8.9029e − 11 \end{matrix}

Example 4

f(x) = e^{x} − \frac{1}{x}

•

g_{0}(x) = x = e^{−x}, g′_{0}(x) = e^{−x}

The iteration from any initial guess x_{0} > 0 will converge to x^{∗} = g_{1}(x^{∗}) = 0.56714.

•

g_{1}(x) = x = − log(x), g′_{1}(x) = 1/x

Around x = x^{∗} = 0.56714, g′_{1}(x) = 1/x > 1, the iteration does not converge.

•

g_{2}(x) = x − \frac{e^{x} − 1/x}{e^{x} + 1/x^{2}} , g′_{2}(x) = \frac{(e^{x} − 1/x)(e^{x} − 2/x^{3})}{(e^{x} + 1/x^{2})^{2}}

Example 5

Consider a 3-variable linear vector function f(x) of arguments x = [x, y, z]^{T}:

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = 6x + 3y + 2z − 18 = 0 \\ f_{2}(x) = 2x + 7y + 3z − 25 = 0 \\ f_{3}(x) = x + 3y + 5z − 22 = 0 \end{matrix}

from which the g-function can be obtained:

g(x) = x,  \{ \begin{matrix}g_{1}(x) = x = −(3y + 2z − 18)/6 \\ g_{2}(x) = y = −(2x + 3z − 25)/7 \\ g_{3}(x) = z = −(x + 3y − 22)/5 \end{matrix}

The Jacobian g(x) of this linear system is a constant matrix

J_{g} = [ \begin{matrix}0 & −1/2 & −1/3 \\ −2/7 & 0 & −3/7 \\ −1/5 & −3/5 & 0 \end{matrix} ]

with the induced p=2 norm (maximum singular value) ||J_{g}|| = 0.851 < 1. Consequently, the iteration x_{n+1} = g(x_{n}) converges from an initial guess x_{0} = [1, 1, 1]^{T} to the solution x = [1, 2, 3]^{T}.

Alternatively, the g-function can also be obtained as

g′(x) = x,  \{ \begin{matrix}g′_{1}(x) = x = −(3y + 5z − 22) \\ g′_{2}(x) = y = −(2x + 3z − 25)/7 \\ g′_{3}(x) = z = −(6x + 3y − 18)/2 \end{matrix}

The Jacobian is

J_{g′} = [ \begin{matrix}0 & −3 & −5 \\ −2/7 & 0 & −3/7 \\ −3 & −3/2 & 0 \end{matrix} ]

with the induced p=2 norm ||J_{g}|| = 5.917 > 1. The iteration does not converge. Example 6 Consider a 3-variable nonlinear function f(x) of arguments x = [x, y, z]^{T}:

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = x^{2} − 2x + y^{2} − z + 1 = 0 \\ f_{2}(x) = xy^{2} − x − 3y + yz + 2 = 0 \\ f_{3}(x) = xz^{2} − 3z + yz^{2} + xy = 0 \end{matrix}

The g-function can be obtained as

g(x) = x,  \{ \begin{matrix}g_{1}(x) = x = (x^{2} + y^{2} − z + 1)/2 \\ g_{2}(x) = y = (xy^{2} − x + yz + 2)/3 \\ g_{3}(x) = z = (xz^{2} + yz^{2} + xy)/3 \end{matrix}

With x_{0} = [0, 0, 0]^{T} and after n > 170 iterations x_{n+1} = g(x_{n}) converges to x_{n} = [1.098933, 0.367621, 0.144932]^{T}, with error ||f(x_{n})|| < 10^{−7}. However, the iteration may not converge from other possible initial guesses.

By Aitken’s method, the iteration x_{n+1} = g(x_{n}) can be accelerated based on two consecutive points x_{0} and x_{1}, as shown in the figure below.

The secant line of g(x) that goes through the two points P_{0} = (x_{0}, g(x_{0}) = x_{1}) and P_{1} = (x_{1}, g(x_{1}) = x_{2}) is represented by the equation in terms of its slope:

\frac{y − g(x_{0})}{x − x_{0}} = \frac{g(x_{1}) − g(x_{0})}{x_{1} − x_{0}}  (1.59)

Solving for y, we get

y = g(x_{0}) + (x − x_{0})\frac{g(x_{1}) − g(x_{0})}{x_{1} − x_{0}} = x_{1} + (x − x_{0})\frac{x_{2} − x_{1}}{x_{1} − x_{0}}  (1.60)

To accelerate, instead of moving from x_{0} to x_{1}, we move to the point x at which this secant line intersects with the identity function y = x. We can therefore replace y in the equation above by x and solve the resulting equation

x = x_{1} + (x − x_{0})\frac{x_{2} − x_{1}}{x_{1} − x_{0}}  (1.61)

to get

x = \frac{x_{1}(x_{1} − x_{0}) − x_{0}(x_{2} − x_{1})}{(x_{1} − x_{0}) − (x_{2} − x_{1})} = x_{0} − \frac{(x_{1} − x_{0})^{2}}{x_{2} − 2x_{1} + x_{0}} = x_{0} − \frac{(Δx_{0})^{2}}{Δ^{2}x_{0}} (1.62)

where Δx_{0} and Δ^{2}x_{0} are respectively the first and second order differences defined below:

Δx_{0} = x_{1} − x_{0}, Δx_{1} = x_{2} − x_{1}  (1.63)

Δ^{2}x_{0} = Δx_{1} − Δx_{0} = (x_{2} − x_{1}) − (x_{1} − x_{0}) = x_{2} − 2x_{1} + x_{0}  (1.64)

This result can then be converted into an iterative process

\{ \begin{matrix}x_{n+1} = g(x_{n}) \\ x_{n+2} = g(x_{n+1}) \\ x_{n+3} = x_{n} − (x_{n+1} − x_{n})^{2}/(x_{n+2} − 2x_{n+1} + x_{n}) \end{matrix}  (1.65)

Given x_{n}, we skip x_{n+1} = g(x_{n}) and x_{n+2} = g(x_{n+1}) but directly move to x_{n+3} computed based on x_{n+1} and x_{n+2}, thereby making a greater step towards the solution.

Example

Solve x^{3}−3x+1 = 0. Construct g(x) = (3x−1)^{1/3}. It takes 18 iterations for the regular fixed point algorithm with initial guess x_{0} = 1, to get x_{18} = 1.5320887 that satisfies |f(x_{n})| < 10^{−6}, but it only three iterations for Aitken’s method to converge to the same result:

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 1.0000 & −1.0000 \\ 1 & 1.259921 & −7.797631e − 01 \\ 2 & 1.406056 & −4.384047e − 01 \\ 3 & 1.476396 & −2.110206e − 01 \\ 4 & 1.507985 & −9.476760e − 02 \\ 5 & 1.521751 & −4.129594e − 02 \\ 6 & 1.527672 & 1.776362e − 02 \\ 7 & 1.530205 & −7.598914e − 03 \\ 8 & 1.531286 & 3.242990e − 03 \\ 9 & 1.531747 & −1.382619e − 03 \\ 10 & 1.531943 & −5.892137e − 04 \\ 11 & 1.532027 & −2.510521e − 04 \\ 12 & 1.532062 & −1.069599e − 04 \\ 13 & 1.532078 & −4.556840e − 05 \\ 14 & 1.532084 & −1.941335e − 05 \\ 15 & 1.532087 & −8.270551e − 06 \\ 16 & 1.532088 & −3.523444e − 06 \\ 17 & 1.532089 & −1.501066e − 06 \\ 18 & 1.532089 & −6.394874e − 07 \end{matrix}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 1.0000 & −1.0000 \\ 1 & 1.5937361 & 2.668730e − 01 \\ 2 & 1.5323992 & 1.254749e − 03 \\ 3 & 1.5320889 & 3.421160e − 08 \end{matrix}

1.4 Newton-Raphson Method (Univariate)

To solve equation f(x) = 0, we first consider the Taylor series expansion of f(x) at any point x_{0}:

f(x) = f(x_{0})+f′(x_{0})(x−x_{0})+ \frac{1}{2!}f′′(x)(x−x_{0})^{2} +⋯+ \frac{1}{n!}f^{(n)}(x_{0})(x−x_{0})^{n} +⋯

(1.66)

If f(x) is linear, i.e., its slope f′(x) is a constant for any x, then the second and higher order terms are all zero, and the equation f(x) = 0 becomes

f(x) = f(x_{0}) + f′(x)(x − x_{0}) = 0  (1.67)

Solving this equation we get the root x^{∗} at which f(x^{∗}) = 0:

x^{∗} = x = x_{0} − \frac{f(x_{0})}{f′(x_{0})} = x_{0} + Δx_{0}  (1.68)

where Δx_{0} = −f(x_{0})/f′(x_{0}) is the step we need to take to go from the initial point x_{0} to the root x^{∗}:

Δx_{0} = −\frac{f(x_{0})}{f′(x_{0})} \{ \begin{matrix}< 0 & if f(x_{0}) and f′(x_{0}) are of the same sign \\ > 0 & if f(x_{0}) and f′(x_{0}) are of different signs \end{matrix}  (1.69)

If f(x) is nonlinear, the sum of the first two terms of the Taylor expansion is only an approximation of f(x), and the resulting x found above can be treated as an approximation of the root, which can be improved iteratively to move from the initial point x_{0} towards the root, as illustrated in the figure below:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − \frac{f(x_{n})}{f′(x_{n})}, n = 0, 1, 2, ⋯  (1.70)

The Newton-Raphson method can be considered as the fixed point iteration x_{n+1} = g(x_{n}) based on

g(x) = x − f(x)/f′(x)  (1.71)

The root x^{∗} at which f(x^{∗}) = 0 is also the fixed point of g(x), i.e., g(x^{∗}) = x^{∗}. For the iteration to converge, g(x) needs to be a contraction with |g′(x)| < 1. Consider

g′(x) = (x − \frac{f(x)}{f′(x)})′ = 1 − \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.72)

We make the following comments:

• At the root x = x^{∗} where f(x^{∗}) = 0, if f′(x^{∗}) ≠ 0, then g′(x^{∗}) = 0 < 1, i.e., g(x) is a contraction and the iteration x_{n+1} = g(x_{n}) converges quadratically when x_{n} is close to x^{∗}.

• If f′(x_{n}) = 0 (the tangent is horizontal), the iteration cannot proceed, but we can modify x_{n} by adding a small value ϵ to x_{n} so that f′(x + ϵ) ≠ 0.

• It is difficult to know the number of roots of a nonlinear equation (unlike the case of a linear equation), which can vary from zero (e.g., f(x) = x^{2} + 1) to infinity (e.g., f(x) = cos(x)). One can try different initial guesses in the range of interest to see if different roots can be found.

• Sometime a parameter δ can be used to control the step size of the iteration:

x_{n+1} = x_{n} − δ \frac{f(x_{n})}{f′(x_{n})}  (1.73)

– If δ < 1, the iteration is de-accelerated. Although the convergence becomes slower, this may be desirable if the function f(x) is not smooth with many local variations.

– If δ > 1, the iteration is accelerated. The convergence may or may not be accelerated. Due to the greater step size, the root may be skipped and missed. Sometimes the convergence may become significantly slowed or even oscillate around the true root, such as the example shown in the figure below with δ = 2.

The order of convergence of the Newton-Raphson iteration can be found based on the Taylor expansion of f(x) at the neighborhood of the root x^{∗} = x_{n} + e_{n}:

0 = f(x^{∗}) = f(x_{n}) + f′(x_{n})e_{n} + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.74)

where e_{n} = x^{∗} − x_{n} is the error at the nth step. Substituting the Newton-Raphson’s iteration

x_{n+1} = x_{n} − \frac{f(x_{n})}{f′(x_{n})}, i.e. f(x_{n}) = f′(x_{n})(x_{n} − x_{n+1})  (1.75)

into the equation above, we get

0 = f′(x_{n})(x_{n} − x_{n+1}) + f′(x_{n})(x^{∗} − x_{n}) + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})
= f′(x_{n})(x^{∗} − x_{n+1}) + \frac{f′′(x)}{2}e_{n}^{2} + O(e_{n}^{3})
= f′(x_{n})e_{n+1} + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.76)

i.e.

e_{n+1} = −\frac{f′′(x_{n})}{2f′(x_{n})}e_{n}^{2} + O(e_{n}^{3})  (1.77)

When n → ∞ all the higher order terms disappear, and the above can be written as

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} ≤ \frac{|f′′(x^{∗})|}{2|f′(x^{∗})|} = μ  (1.78)

Alternatively, we can get the Taylor expansion in terms of g(x):

x_{n+1} = x^{∗} − e_{n+1} = g(x_{n}) = g(x^{∗} − e_{n}) = g(x^{∗}) − g′(x^{∗})e_{n} + \frac{g′′(x^{∗})}{2}e_{n}^{2} + O(e_{n}^{3})

(1.79)

Subtracting g(x^{∗}) = x^{∗} from both sides we get:

e_{n+1} = g′(x^{∗})e_{n} − \frac{g′′(x^{∗})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.80)

Now we find g′(x) and g′′(x):

g′(x) = (x − \frac{f(x)}{f′(x)})′ = 1 − \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.81)

and

g′′(x) = (\frac{f(x)f′′(x)}{(f′(x))^{2}})′ = \frac{(f′(x)f′′(x) + f(x)f′′′(x))(f′(x))^{2} − f(x)f′′(x)2f′(x)f′′(x)}{(f′(x))^{4}}
= \frac{(f′(x))^{3}f′′(x)}{(f′(x))^{4}} = \frac{f′′(x)}{f′(x)}  (1.82)

Evaluating these at x = x^{∗} at which f(x^{∗}) = 0, and substituting them back into the expression for e_{n+1} above, the linear term is zero as g′(x^{∗}) = 0, i.e., the convergence is quadratic, and we get the same result:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} ≤ \frac{|f′′(x^{∗})|}{2|f′(x^{∗})|} = \frac{|g′′(x^{∗})|}{2}  (1.83)

We see that, if f′(x^{∗}) ≠ 0, then the order of convergence of the Newton-Raphson method is p = 2, and the rate of convergence is μ = |f′′(x)|/2|f′(x)|. However, if f′(x^{∗}) = 0, the convergence is linear rather than quadratic, as shown in the example below.

Example: Consider solving the equation

f(x) = x^{3} − 4x^{2} + 5x − 2 = (x − 1)^{2}(x − 2) = 0  (1.84)

which has a repeated root x = 1 as well as a single root x = 2. We have

f′(x) = 3x^{2} − 8x + 5 = (3x − 5)(x − 1),  f′′(x) = 6x − 8  (1.85)

Note that at the root x^{∗} = 1 we have f′(x^{∗}) = f′(1) = 0. We further find:

g(x) = x − \frac{f(x)}{f′(x)} = x − \frac{(x − 1)^{2}(x − 2)}{(3x − 5)(x − 1)} = x − \frac{(x − 1)(x − 2)}{3x − 5}  (1.86)

and

g′(x) = \frac{f(x)f′′(x)}{(f′(x))^{2}} = \frac{(x − 1)^{2}(x − 2)(6x − 8)}{(3x − 5)^{2}(x − 1)^{2}}
= \frac{(x − 2)(6x − 8)}{(3x − 5)^{2}} = \{ \begin{matrix}1/2 & x = 1 \\ 0 & x = 2 \end{matrix}

We therefore have

e_{n+1} = g′(x)e_{n} − \frac{1}{2}g′′(x)e_{n}^{2} + O(e_{n}^{3}) ⟹^{n→∞} \{ \begin{matrix}e_{n}/2 & x = 1 \\ −g′′(x)e_{n}^{2}/2 & x = 2 \end{matrix}  (1.87)

We see the iteration converges quadratically to the single root x = 2, but only linearly to the repeated root x = 1.

We consider in general a function with a repeated root at x = a of multiplicity k:

f(x) = (x − a)^{k}h(x)  (1.88)

its derivative is

f′(x) = k(x−a)^{k−1}h(x)+(x−a)^{k}h′(x) = (x−a)^{k−1}[kh(x)+(x−a)h′(x)] (1.89)

As f′(a) = 0, the convergence of the Newton-Raphson method to x = a is linear, rather than quadratic.

In such case, we can accelerate the iteration by using a step size δ = k > 1:

g(x) = x − k \frac{f(x)}{f′(x)}  (1.90)

and

g′(x) = 1 − k \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = 1 − k + k \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.91)

Now we show that this g′(x) is zero at the repeated root x = a, therefore the convergence to this root is still quadratic.

We substitute f(x) = (x − a)^{k}h(x), f′(x) = k(x − a)^{k−1}[kh + (x − a)h′], and

f′′(x) = [(x − a)^{k−1}[kh + (x − a)h′]]′
= 1 − k + k (k − 1)(x − a)^{k−2}[kh + (x − a)h′]
+(x − a)^{k−1}[(k + 1)h′ + (x − a)h′′]  (1.92)

into the expression for g′(x) above to get (after some algebra):

g′(x) = 1 − k + k \frac{f(x)f′′(x)}{(f′(x))^{2}}
= 1 − k + k \frac{h(k − 1)(kh + (x − a)h′) + h(x − a)((k + 1)h′ + (x − a)h′)}{(kh + (x − a)h′)^{2}} (1.93)

At x = a, we get

g′(x)|_{x=a} = 1 − k + k \frac{(k − 1)kh^{2}}{(kh)^{2}} = 0  (1.94)

i.e., the convergence to the repeated root at x = a is no longer linear but quadratic.

The difficulty, however, is that the multiplicity k of a root is unknown ahead of time. If δ > 1 is used blindly some root may be skipped, and the iteration may oscillate around the real root.

Example: Consider solving f(x) = x^{3} − 4x^{2} + 5x − 2 = (x − 1)^{2}(x − 2) = 0, with a double root x = 1 and a single root x = 2. In the following, we compare the performance of both g_{1}(x) = x − f(x)/f′(x) and g_{2}(x) = x − 2f(x)/f′(x).

• First use an initial guess x_{0} = 0.3.

– When g_{1}(x) is used, the convergence is linear around the double root x = 1. It takes 16 iterations to get x_{16} = 0.999983 with |e_{16}| < 10^{−9}:

n  x_{n}  f(x_{n})  f′(x_{n})
0 0.300000 −0.833000 2.870000
1 0.590244 −0.236698 1.323212
2 0.769125 −0.065609 0.621659
3 0.874665 −0.017678 0.297797
4 0.934027 −0.004640 0.145004
5 0.966023 −0.001194 0.071417
6 0.982737 −0.000303 0.035420
7 0.991296 −0.000076 0.017636
8 0.995629 −0.000019 0.008799
9 0.997810 −0.000005 0.004395
10 0.998904 −0.000001 0.002196
11 0.999452 −0.000000 0.001098
12 0.999726 −0.000000 0.000549
13 0.999863 −0.000000 0.000274
14 0.999931 −0.000000 0.000137
15 0.999966 −0.000000 0.000069
16 0.999983 −0.000000

– When g_{2}(x) is used, the convergence is quadratic around the double root x = 1. It takes only 3 iterations to get x_{3} = 0.999982 with |e_{3}| < 10^{−9}:

n  x_{n}  f(x_{n})  f′(x_{n})
0 0.300000 −0.833000 2.870000
1 0.880488 −0.015990 0.281874
2 0.993944 −0.000037 0.012222
3 0.999982 −0.000000

• Next use a different initial guess x_{0} = 4.

– If g_{1} is used, it takes 7 iterations to get x_{7} = 2.0 with |e_{7}| < 10^{−9}, the convergence is quadratic.

n  x_{n}  f(x_{n}) f′(x_{n})
1 x = 3.00000 4.00000 8.00000
2 x = 2.50000 1.12500 3.75000
3 x = 2.20000 0.28800 1.92000
4 x = 2.05000 0.05513 1.20750
5 x = 2.00435 0.00439 1.01745
6 x = 2.00004 0.00004 1.00015
7 x = 2.00000 0.00000 1.00000
8 x = 2.00000 0.00000

– If g_{2} is used, oscillation happens as shown in the figure below. However, if a better initial guess x_{0} = 3.5 is used instead of x_{0} = 4, it takes only one step to get x_{1} = 2.0 with |e_{1}| < 10^{−9}, the convergence is significantly accelerated.

n  x_{n}  f(x_{n}) f′(x_{n})
1 3.00000 4.00000 8.00000
2 2.00000 0.00000

1.5 Newton-Raphson Method (Multivariate)

The Newton-Raphson method discussed above for solving a single-variable equation f(x) = 0 can be generalized to the case of multivariate equation systems containing M equations of N variables in x = [x_{1}, ⋯, x_{N}]^{T}:

f(x) = [ f_{1}(x) ; ⋮ ; f_{M}(x) ] = [ f_{1}(x_{1}, ⋯, x_{N}) ; ⋮ ; f_{M}(x_{1}, ⋯, x_{N}) ] = [ 0 ⋮ 0 ] = 0  (1.95)

To solve the equation system, we first consider the Taylor series expansion of each of the M functions in the neighborhood of the initial point x_{0} = [x_{01}, ⋯, x_{0N}]^{T}:

f_{m}(x) = f_{m}(x_{0})+ ∑_{n=1}^{N} \frac{∂f_{m}(x_{0})}{∂x_{n}}(x_{n} −x_{0n})+r_{m}(||x−x_{0}||^{2}), (m = 1, ⋯, M)

(1.96)

where r_{m}(||x − x_{0}||^{2}) represents the second and higher order terms in the series beyond the linear term, which can be neglected if ||x − x_{0}|| is small. These M equations can be expressed in matrix form

[ f_{1}(x) ; ⋮ ; f_{M}(x) ] = [ f_{1}(x_{0}) ; ⋮ ; f_{M}(x_{0}) ]+[ \begin{matrix}\frac{∂f_{1}(x_{0})}{∂x_{1}} & ⋯ & \frac{∂f_{1}(x_{0})}{∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{M} (x_{0})}{∂x_{1}} & ⋯ & \frac{∂f_{M} (x_{0})}{∂x_{N}} \end{matrix} ] [ x_{1} − x_{01} ; ⋮ ; x_{N} − x_{0N} ]+[ r_{1} ; ⋮ ; r_{M} ]

(1.97)

or more concisely

f(x) = f(x_{0}) + J(x_{0}) (x − x_{0}) + r ≈ f(x_{0}) + J(x_{0}) (x − x_{0}) = f_{0} + J_{0} Δx (1.98)

where Δx = x − x_{0}, and f_{0} = f(x_{0}) and J_{0} = J(x_{0}) are respectively the function f(x) and its Jacobian matrix J_{f}(x) both evaluated at x_{0}. We further consider solving the equation system f(x) = 0 in the following two cases:

• M = N: The number of equations is the same as the number of unknowns, the Jacobian J(x) is a square matrix and its inverse J^{−1} exists in general. In the special case where f(x) is linear, the Taylor series contains only the first two terms while all higher order terms are zero, and the approximation in Eq. (1.98) becomes exact. To find the root x^{∗} satisfying f(x^{∗}) = 0, we set f(x) in Eq. (1.98) to zero and solve the resulting equation for x to get

x^{∗} = x = x_{0} − J_{0}^{−1}f_{0} = x_{0} − A^{−1}f_{0}  (1.99)

As in general f(x) is nonlinear, it can only be approximated by the first two terms of the Taylor series, consequently the result above is only an approximation of the optimal solution. But this approximation can be improved iteratiively to approach the optimal solution x^{∗}:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − J_{n}^{−1} f_{n}  (1.100)

where we have defined Δx_{n} = −J_{n}^{−1} f_{n} as the increment, which can also be denoted by d_{n} = Δx_{n} to represent the search or Newton direction. The iteration moves x_{n} in the N-D space spanned by {x_{1}, ⋯, x_{N}} from some initial guess x_{0} along such a path that all function values f_{m}(x), m = 1, ⋯, M) are reduced. Same as in the univariate case, a scaling factor δ_{n} can be used to control the step size of the iteration

x_{n+1} = x_{n} + δ_{n} Δx_{n} = x_{n} − δ_{n} J_{n}^{−1} f_{n}  (1.101)

When δ < 1, the step size becomes smaller and the convergence of the iteration is slower, however, we will have a better chance not to skip a solution, which may happen if f(x) is not smooth and the step size is too big.

The algorithm is listed below:

– Select x_{0}

– Obtain f_{0} = f(x_{0}) and J_{0}

– Obtain J_{0}^{−1}

– n = 0

– While ||x_{n+1} − x_{n}|| > tol do ∘ x_{n+1} = x_{n} − J_{n}^{−1}f_{n} ∘ Find f_{n+1} and J_{n+1} ∘ n = n + 1

• M > N: There are more equations than unknowns, i.e., equation f(x) = 0 is an over-constrained system, and the Jacobian J(x) is an M × N nonsquare matrix without an inverse, i.e., no solution exists for the equation f(x) = 0 in general. But we can still seek to find an optimal solution x^{∗} that minimizes the following sum-of-squares error:

ε(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2}f(x)^{T} f(x) = \frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)  (1.102)

The gradient vector of ε(x) is:

g_{ε}(x) = \frac{d}{dx}ε(x) = \frac{d}{dx} (\frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)) = ∑_{m=1}^{M} \frac{d}{dx}f_{m}(x) f_{m}(x) (1.103)

The nth component of g_{ε}(x) is

\frac{∂ε(x)}{∂x_{n}} = ∑_{m=1}^{M} \frac{∂f_{m}(x)}{∂x_{n}} f_{m}(x) = ∑_{m=1}^{M} J_{mn} f_{m}(x) (n = 1, ⋯, N) (1.104)

where J_{mn} = ∂f_{m}(x)/∂x_{n} is the component in the mth row and nth column of the Jacobian matrix J_{f}(x) of f(x). Now the gradient can be written as

g_{ε}(x) = J_{f}^{T} (x) f(x)  (1.105)

If f(x) is linear and can therefore be represented as the sum of the first two terms of its Taylor series in Eq. (1.98), then the gradient is:

g_{ε}(x) = J^{T} (x) f(x) = J^{T} (x_{0}) [f(x_{0}) + J(x_{0})Δx] = J_{0}^{T} (f_{0} + J_{0}Δx) (1.106)

where x_{0} is any chosen initial guess. If we assume x is the optimal solution at which ε(x) is minimized and g_{ε}(x) is zero:

g_{ε}(x) = J_{0}^{T} (f_{0} + J_{0}Δx) = 0  (1.107)

then the increment Δx can be found by solving the equation

Δx = −(J_{0}^{T} J_{0})^{−1}J_{0}^{T} f_{0} = −J_{0}^{−} f_{0}  (1.108)

Here J_{0}^{−} = (J_{0}^{T} J_{0})^{−1}J_{0}^{T} is the pseudo-inverse (Section A.15) of the nonsquare matrix J_{0}. Now the optimal solution can be found as:

x^{∗} = x_{0} + Δx = x_{0} − J_{0}^{−} f_{0}  (1.109)

However, as f(x) is nonlinear in general, the sum of the first two terms of its Taylor series is only an approximation. Consequently the result x = x_{0}+Δx above is not the optimal solution, but an estimate which can be improved by carrying out this step iteratively:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − J_{n}^{−} f_{n}  (1.110)

This iteration will converge to x^{∗} at which g_{ε}(x^{∗}) = 0, and the squared error ε(x) is minimized.

Specially, for a linear equation system f(x) = Ax − b = 0, the Jacobian is simplely J_{f}(x_{n}) = A, and the optimal solution is

x_{n+1} = x_{n}−A^{−}f_{n} = x_{n}−(A^{T} A)^{−1}A^{T} (Ax_{n}−b) = (A^{T} A)^{−1}A^{T} b = A^{−}b

(1.111)

i.e., the optimal solution can be found from any initial guess in a single step. This result is the same as that in Eq. (1.7).

Comparing Eqs. (1.100) and (1.110), we see that the two algorithms are essentially the same, with the only difference that the regular inverse J^{−1} is used when M = N, but the pseudoinverse J^{−} is used when M > N and J^{−1} does not exist.

The Newton-Raphson method assumes the availability of the analytical expressions of all partial derivatives J_{mn} = ∂f_{m}(x)/∂x_{n} (m = 1, ⋯, M, n =

1.5 Newton-Raphson Method (Multivariate) 29

1, ⋯, N) in the Jacobian matrix J. However, when this is not the case, J_{mn} need to be approximated by forward or central difference (secant) method:

J_{mn} = \frac{∂f_{m}(x_{1}, ⋯, x_{N})}{∂x_{n}} ≈ \frac{f_{m}(x_{1}, ⋯, x_{n} + h, ⋯, x_{N}) − f_{m}(x_{1}, ⋯, x_{n}, ⋯, x_{N})}{h}
≈ \frac{f_{m}(x_{1}, ⋯, x_{n} + h, ⋯, x_{N}) − f_{m}(x_{1}, ⋯, x_{n} − h, ⋯, x_{N})}{2h}  (1.112)

where h is a small increment.

Example 1

\{ \begin{matrix}3 x_{1} − cos(x_{2}x_{3}) − 3/2 = 0 \\ 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{3} − 1 = 0 \\ 20 x_{3} + e^{−x_{1}x_{2}} + 9 = 0 \end{matrix}

J = [ \begin{matrix}3 & x_{3} sin(x_{2}x_{3}) & x_{2} sin(x_{2}x_{3}) \\ 8 x_{1} & −1250 x_{2} & 2 \\ −x_{2} e^{−x_{1}x_{2}} & −x_{1} e^{−x_{1}x_{2}} & 20 \end{matrix} ]

\begin{matrix}n & x & error \\ 0 & (1.000000, 1.000000, 1.000000) & 6.207e + 02 \\ 1 & (1.232701, 0.503132, −0.473253) & 1.541e + 02 \\ 2 & (0.832592, 0.251806, −0.490636) & 3.884e + 01 \\ 3 & (0.833238, 0.128406, −0.494702) & 9.517e + 00 \\ 4 & (0.833275, 0.069082, −0.497147) & 2.200e + 00 \\ 5 & (0.833281, 0.043585, −0.498206) & 4.063e − 01 \\ 6 & (0.833282, 0.036117, −0.498517) & 3.486e − 02 \\ 7 & (0.833282, 0.035343, −0.498549) & 3.741e − 04 \\ 8 & (0.833282, 0.035335, −0.498549) & 4.498e − 08 \\ 9 & (0.833282, 0.035335, −0.498549) & 5.551e − 16 \end{matrix}

Example 2

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = x_{1}^{2} − 2x_{1} + x_{2}^{2} − x_{3} + 1 = 0 \\ f_{2}(x) = x_{1}x_{2}^{2} − x_{1} − 3x_{2} + x_{2}x_{3} + 2 = 0 \\ f_{3}(x) = x_{1}x_{3}^{2} − 3x_{3} + x_{2}x_{3}^{2} + x_{1}x_{2} = 0 \end{matrix}

J = [ \begin{matrix}2x_{1} − 2 & 2x_{2} & −1 \\ x_{2}^{2} − 1 & 2x_{1}x_{2} − 3 + x_{3} & x_{2} \\ x_{3}^{2} + x_{2} & x_{3}^{2} + x_{1} & 2x_{1}x_{3} − 3 + 2x_{2}x_{3} \end{matrix} ]

With x_{0} = [1, 2, 3]^{T}, we get a root:

\begin{matrix}n & x & error \\ 0 & (1.00000, 2.00000, 3.00000) & 2.064e + 01 \\ 1 & (0.10256, 1.64103, 2.56410) & 4.303e + 00 \\ 2 & (1.52062, 1.41113, 0.19859) & 2.689e + 00 \\ 3 & (1.94123, 0.77134, 0.89465) & 1.217e + 00 \\ 4 & (1.06737, 1.19117, 0.48353) & 1.144e + 00 \\ 5 & (1.26825, 0.95182, 0.88028) & 3.323e − 01 \\ 6 & (0.95899, 1.03384, 0.96813) & 1.171e − 01 \\ 7 & (1.00171, 1.00007, 0.99718) & 4.162e − 03 \\ 8 & (1.00000, 1.00000, 1.00000) & 6.701e − 06 \end{matrix}

With x_{0} = [0, 0, 0]^{T}, we get another root:

\begin{matrix}n & x & error \\ 0 & (0.00000, 0.00000, 0.00000) & 2.236e + 00 \\ 1 & (0.50000, 0.50000, 0.00000) & 5.728e − 01 \\ 2 & (0.83951, 0.47531, 0.13580) & 1.175e − 01 \\ 3 & (0.98582, 0.41849, 0.15069) & 2.639e − 02 \\ 4 & (1.05417, 0.38715, 0.14717) & 6.088e − 03 \\ 5 & (1.08565, 0.37339, 0.14558) & 1.264e − 03 \\ 6 & (1.09693, 0.36849, 0.14503) & 1.618e − 04 \\ 7 & (1.09888, 0.36764, 0.14494) & 4.817e − 06 \end{matrix}

Broyden’s method

In the Newton-Raphson method, two main operations are carried out in each iteration: (a) evaluate the Jacobian matrix J_{f} (x_{n}) and (b) obtain its inverse J_{f}^{−1}(x_{n}). To avoid these expensive computation for these operations, we can consider using Broyden’s method, one of the quasi-Newton methods, which approximates the inverse of the Jacobian J_{n+1}^{−1} = J^{−1}(x_{n+1}) from the J_{n}^{−1} = J^{−1}(x_{n}) in the previous iteration step, so that it can be updated iteratively from the initial J_{0}^{−1} = J^{−1}(x_{0}).

We first consider in the single-variable case how to estimate the next derivative f′_{n+1} = f′(x_{n+1}) from the current f′_{n} = f′(x_{n}) by the secant method:

f′_{n+1} ≈ f̂′_{n+1} = \frac{f_{n+1} − f_{n}}{x_{n+1} − x_{n}} = \frac{δf_{n}}{δx_{n}} = \frac{f̂′_{n}δx_{n} − f̂′_{n}δx_{n} + δf_{n}}{δx_{n}}
= f̂′_{n} + \frac{δf_{n} − f̂′_{n}δx_{n}}{δx_{n}} = f̂′_{n} + δ̂f′_{n}  (1.113)

where

• δf_{n} = f_{n+1} − f_{n} is the true increment of the function over the interval δx_{n} = x_{n+1} − x_{n}; • f̂′_{n}δx_{n} is the estimated increment of the function based on the previous derivate

f̂′_{n};

• δ̂f′_{n} is the estimated increment of the derivative:

δ̂f′_{n} = \frac{δf_{n} − f̂′_{n}δx_{n}}{δx_{n}}  (1.114)

The equation above indicates that the derivative f′_{n+1} in the (n + 1)th step can be estimated by adding the estimated increment δ̂f′_{n} to the derivative f̂′n in the current nth step.

Having obtained f̂′_{n+1}, we can use the same iteration in the Newton-Raphson method to find x_{n+1}:

x_{n+1} = x_{n} − \frac{f(x_{n})}{f̂′_{n}}  (1.115)

This method for single-variable case can be generalized to multiple variable case for solving f(x) = 0. Following the way we estimate the increment of the derivative of a single-variable function in Eq. (1.114), here we can estimate the increment of the Jacobian of a multi-variable function:

δĴ_{n} = \frac{(δf_{n} − Ĵ_{n}δx_{n}) δx_{n}^{T}}{δx_{n}^{T} δx_{n}} = \frac{(δf_{n} − Ĵ_{n}δx_{n})δx_{n}^{T}}{||δx_{n}||^{2}}  (1.116)

where δx_{n} = x_{n+1} − x_{n} and δf_{n} = f_{n+1} − f_{n} = f(x_{n+1}) − f(x_{n}). Now in each iteration, we can update the estimated Jacobian as well as the estimated root:

Ĵ_{n+1} = Ĵ_{n} + δĴ_{n},  x_{n+1} = x_{n} + δx_{n} = x_{n} − Ĵ_{n}^{−1}f(x_{n})  (1.117)

The algorithm can be further improved so that the inverse of the Jacobian J_{n} is avoided. Specifically, consider the inverse Jacobian:

Ĵ_{n+1}^{−1} = (Ĵ_{n} + δĴ_{n})^{−1} = [Ĵ_{n} + \frac{(δf_{n} − Ĵ_{n}δx_{n})δx_{n}^{T}}{||δx_{n}||^{2}} ]^{−1}  (1.118)

We can apply the Sherman-Morrison formula (Section A.6):

(A + uv^{T})^{−1} = A^{−1} − \frac{A^{−1}uv^{T} A^{−1}}{1 + v^{T} A^{−1}u}  (1.119)

to the right-hand side of the equation above by defining

A = Ĵ_{n},  u = \frac{δf_{n} − Ĵ_{n}δx_{n}}{||δx_{n}||^{2}} ,  v = δx_{n}  (1.120)

and rewrite it as:

Ĵ_{n+1}^{−1} = Ĵ_{n}^{−1} − \frac{Ĵ_{n}^{−1} (δf_{n} − Ĵ_{n}δx_{n}^{T})/||δx_{n}||^{2} δx_{n}^{T} Ĵ_{n}^{−1}}{1 + δx_{n}^{T} Ĵ_{n}^{−1}(δf_{n} − Ĵ_{n}δx_{n})/||δx_{n}||^{2}} = Ĵ_{n}^{−1} − \frac{(Ĵ_{n}^{−1}δf_{n} − δx_{n}) δx_{n}^{T} Ĵ_{n}^{−1}}{δx_{n}^{T} Ĵ_{n}^{−1}δf_{n}}

(1.121)

We see that the next Ĵ_{n+1}^{−1} can be iteratively estimated directly from the previous Ĵ_{n}^{−1}, thereby avoiding computing the inverse of Ĵ_{n} altogether. The algorithm is listed below:

• Select x_{0}

• Find f_{0} = f(x_{0}), J_{0} = J(x_{0}), and J_{0}^{−1} • x_{1} = x_{0} − J_{0}^{−1}f_{0}

• n = 0

• While ||x_{n+1} − x_{n}|| > tol do – δx_{n} = x_{n+1} − x_{n}

– f_{n+1} = f(x_{n+1})

– δf = f_{n+1} − f_{n}

– J_{n}^{−1} = J_{n}^{−1} − (δx_{n} − J_{n}^{−1}δf_{n})δx_{n}^{T} J_{n}^{−1}/(δx_{n}^{T} J_{n}^{−1}δf_{n})

– x_{n} = x_{n+1}, f_{n} = f_{n+1}

– x_{n+1} = x_{n} − J_{n}^{−1}f_{n}

– n = n + 1

2 Optimization

Machine learning problems can often be formulated as an optimization problem. In many algorithms for both regression and classification, the goal is to optimize an objective function of a set of model parameters so that the model fits the given dataset optimally in certain sense.

Consider, for example, the construction of a model parameterized by θ = [θ_{1}, ⋯, θ_{N}]^{T} to fit a set of observed data points called the training dataset or simply training set denoted by D = {(x_{n}, y_{n}), n = 1, ⋯, N}. While the form of the model is typically assumed to be known (e.g., a linear or quadratic function, or certain probability density function (pdf) of the data), the parameters in θ are unknown and to be estimated based on the training set D, using methods such as the following widely used in machin learning:

• Least squares estimation (LSE):

The sum of squared error (SSE) between the model output and the desired output based on the given data is to be minimized:

J(θ) = ∑_{n=1}^{N}[y_{n} − f((x_{n}, θ))]^{2}  (2.1)

• Maximum likelihood estimate (MLE):

The likelihood function of the parameters in θ is to be maximized:

J(θ) = L(θ|D) ∝ p(D|θ) = ∏_{n=1}^{N} p(y_{n}|x_{n}, θ)  (2.2)

Solving these optimization problems we get the optimal model parameter θ^{∗} for the model to best fit the given data.

To prepare for the discussions of these algorithms in the future chapters, here in this chapter we consider a set of optimization algorithms for optimizing a generic real multivariate function y = f(x) = f(x_{1}, ⋯, x_{N}) by searching its domain, the N-D space spanned by {x_{1}, ⋯, x_{N}}, to find the optimal solution x^{∗} = [x_{1}^{2}, ⋯, x_{N}^{∗}]^{T} at which the function value f(x^{∗}) is either maximized or minimized. As maximizing f(x) is equivalent to minimizing −f(x), we typically consider only minimization problems to find:

x^{∗} = arg min_{x} f(x), i.e., f(x^{∗}) = min_{x} f(x)  (2.3)

If the independent variable x is allowed to take any value inside the function domain ℝ^{N}, then the optimization problem is unconstrained, otherwise if x must be inside a subset of ℝ^{N}, the problem is constrained. Both types of optimization problems occuur in machine learning.

2.1 Unconstrained Optimization

2.1.1 Optimization by Solving Equations

Before discussing specific optimizationalgorithms, we first we show that the problems of unconstrained minimization and equation solving are equivalent as they can be converted to each other.

• To minimize an objective function f(x), we can set its gradient vector g_{f}(x) to zero and solve the resulting equation system:

g_{f}(x) = ▽_{x}f(x) = \frac{df(x)}{dx} = [ ∂f/∂x_{1} ; ⋮ ; ∂f/∂x_{N} ] = 0  (2.4)

The solution x^{∗} of this quation system, called a critical, stationary, or stable point of f(x), is also the solution to the optimization problem, if it is not a saddle point, that either maximizes or minimizes f(x).

This equation system can be solved by any of the methods discussed in the previous chapter, such as the Newton-Raphson method, which finds the root of a general function f(x) iteratively: x_{n+1} = x_{n} − J_{f}^{−1}(x_{n}) f(x_{n}). Here, specifically, to solve the equation g_{f}(x) = 0, we first get the Jacobian J_{g}(x) = H_{f}(x) of the gradient g_{f}(x) of f(x), which is the Hessian of f(x), and then carry out the iteration below to eventually find x^{∗} that minimizes f(x):

x_{n+1} = x_{n} − J_{g}^{−1}(x_{n}) g_{f}(x_{n}) = x_{n} − H_{f}^{−1}(x_{n}) g_{f}(x_{n})  (2.5)

The second equality is due to the fact that the Jacobian of the gradient g_{f} of function f(x) is the Hessian of the function, i.e., J_{g} = H_{f}.

• To solve an equation system f(x) = [f_{1}(x), ⋯, f_{N}(x)]^{T} = 0, we can minimize an objective function defined as

J(x) = \frac{1}{2}f^{T} (x)f(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2} ∑_{i=1}^{N} |f_{i}(x)|^{2}  (2.6)

The solution that minimizes J(x) is also the solution of the original equation. To see this, we again set the gradient of J(x) zero to get as above:

g_{J}(x) = ▽_{x}J(x) = \frac{d}{dx} [\frac{1}{2}f^{T} (x)f(x)] = f′(x) f(x) = J_{f} (x) f(x) = 0

(2.7)

As in general the Jicobian J_{f} (x) = f′(x) is a full rank matrix, the homogeneous equation above only has a zero solution f(x) = 0, which is indeed the original equation.

2.1.2 Newton’s method

We first consider Newton’s method applied to minimizing a univariate function f(x). The Taylor series expansion (Section A.22) of the function f(x) around a point x_{0} is:

f(x) = f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2} + ⋯ + \frac{1}{n!}f^{(n)}(x_{0})(x − x_{0})^{n} + ⋯
≈ f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2} = q(x)  (2.8)

where the first and second order derivatives f′(x) and f′′(x) represent respectively the local change and the more global curvature of the function.

If f(x) is a quadratic function denoted by q(x), then its Taylor series contains only the first three terms (constant, linear, and quadratic terms). In this case, we can find the vertex point at which f(x) = q(x) reaches its extremum, by first setting its derivative to zero:

q′(x) = \frac{d}{dx}q(x) = \frac{d}{dx} [f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2}]
= f′(x_{0}) + f′′(x_{0})(x − x_{0}) = 0  (2.9)

and then solving the resulting equation to get:

x^{∗} = x = x_{0} − \frac{f′(x_{0})}{f′′(x_{0})} = x_{0} + Δx_{0}  (2.10)

where Δx_{0} = −f′(x_{0})/f′′(x_{0}) is the step we need to take to move from any initial point x_{0} to the solution x^{∗} in a single step. Note that f(x^{∗}) is a minimum if f′′(x^{∗}) > 0 but a maximum if f′′(x^{∗}) < 0.

If f(x) ≠ q(x) is not quadratic, the result above can still be considered as an approximation of the solution, which can be improved iteratively from the initial guess x_{0} to eventually approach the solution:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − \frac{f′(x_{n})}{f′′(x_{n})}, n = 0, 1, 2, ⋯  (2.11)

where Δx_{n} = −f′(x_{n})/f′′(x_{n}). We see that in each step x_{n} of the iteration, the function f(x) is fitted by a quadratic functions q(x_{n}) and its vertex at x_{n+1} is used as the updated approximated solution, at which the function is again fitted by another quadratic function q(x_{n+1}) for the next iteration. Through this process the solution can be approached as n → ∞.

We note that the iteration x_{n+1} = x_{n} − f′(x_{n})/f′′(x_{n}) is just Eq. (1.18) in Chapter 1, the Newton-Raphson method applied to solving equation f′(x) = 0, as the necessary condition for an extremum of f(x).

Newton’s method for the minimization of a single-variable function f(x) can be generalized for the minimization of a mult-variable function f(x) = f(x_{1}, ⋯, x_{N}). Again, we approximate f(x) by a quadratic function q(x) containing only the first three terms of its Taylor series at some initial point x_{0}:

f(x) ≈ f(x_{0}) + g_{0}^{T} (x − x_{0}) + \frac{1}{2}(x − x_{0})^{T} H_{0} (x − x_{0}) = q(x)  (2.12)

where g_{0} and H_{0} are respectively the gradient vector and Hessian matrix of the function f(x) at x_{0}:

g_{0} = g_{f}(x_{0}) = \frac{d}{dx}f(x_{0}) = [ ⋮_{\frac{∂f(x_{0})}{∂x_{N}}}^{\frac{∂f(x_{0})}{∂x_{1}}} ] ,

H_{0} = H_{f}(x_{0}) = \frac{d}{dx}g(x_{0}) = \frac{d^{2}}{dx^{2}} f(x_{0}) = [ \begin{matrix}\frac{∂^{2}f(x_{0})}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x_{0})}{∂x_{1}∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x_{0})}{∂x_{N} ∂x_{1}} & ⋯ & \frac{∂^{2}f(x_{0})}{∂x_{N}^{2}} \end{matrix} ](2.13)

The stationary point of f(x) = q(x) can be found from any initial guess x_{0} by setting its derivative to zero:

\frac{d}{dx}q(x) = \frac{d}{dx} [f(x_{0}) + g_{0}^{T} (x − x_{0}) + \frac{1}{2}(x − x_{0})^{T} H_{0} (x − x_{0})]
= g_{0} + H_{0} (x − x_{0}) = 0  (2.14)

and solving the resulting equation to get

x^{∗} = x = x_{0} − H(x_{0})^{−1}g(x_{0})  (2.15)

Similar to the single-variable case in which f(x^{∗}) is either a minimum or maximum depends on whether the second order derivative f′′(x^{∗}) is greater or smaller than zero, here in multi-variable case the function value f(x^{∗}) at the stationary point is a maximum, minimum, or saddle point, depending on the curvature of the function represented by the Hessian matrix H^{∗} = H^{∗} = H_{f}(x)|_{x=x^{∗}} evaluated at x^{∗}:

• If H^{∗} > 0 is positive definite (all eigenvalues are positive), f(x^{∗}) is a minimum;

• If H^{∗} < 0 is negative definite (all eigenvalues are negative), f(x^{∗}) is a maximum;

• If H^{∗} is indefinite (with both positive and negative eigenvalues), f(x^{∗}) is a saddle point (maximum in some directions, but minimum in others).

If f(x) ≠ q(x) is not quadratic, the result above is not a minimum or maximum, but it can still be used as an approximation of the solution, which can be improved to approach the solution iteratively:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − H_{n}^{−1}g_{n} = x_{n} + d_{n}, n = 0, 1, 2, ⋯  (2.16)

where g_{n} = g(x_{n}), H_{n} = H(x_{n}), and the increment d_{n} = Δx_{n} = −H_{n}^{−1}g_{n} is called Newton search direction. We note that this iteration is just a generalization of x_{n+1} = x_{n} − f′(x_{n})/f′′(x_{n}) in 1-D case where g_{n} = f′(x_{n}) and H_{n} = f′′(x_{n}), and the iteration x_{n+1} = x_{n} − H_{n}^{−1} g_{n} above is just Eq. (1.100) in Chapter 1, the Newton-Raphson method applied to solving equation f′(x) = g_{f}(x) = 0, as the necessary condition for an extremum of f(x). The computational complexity for each iteration is O(N^{3}) due to the inverse operation H_{n}^{−1}.

Example 1:

An over-constrained nonlinear equation system f(x) = 0 of M equations and N < M unknowns can be solved by minimizing the following SSE:

ε(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)  (2.17)

To use the Newton-Raphson method, we first find the gradient of the error function:

g_{ε}(x) = \frac{d}{dx}ε(x) = \frac{1}{2} \frac{d}{dx}||f(x)||^{2} = \frac{df}{dx} f = J_{f}^{T} f  (2.18)

of which the ith component is:

g_{i} = \frac{∂}{∂x_{i}} (\frac{1}{2}||f||^{2}) = \frac{1}{2} ∑_{m=1}^{M} \frac{∂}{∂x_{i}} f_{m}^{2} = ∑_{m=1}^{M} \frac{∂f_{m}}{∂x_{i}} f_{m} = ∑_{m=1}^{M} J_{mi} f_{m} (2.19)

where J_{mi} = ∂f_{m}/∂x_{i} is the component of the function’s Jacobian J_{f}(x) in the mth row and ith column.

We further find the component of the Hessian H_{ε} of the error function in the ith row and jth column:

H_{ij} = \frac{∂^{2}}{∂x_{i}∂x_{j}} (\frac{1}{2}||f||^{2}) = \frac{∂}{∂x_{j}} g_{i} = ∑_{m=1}^{M} \frac{∂}{∂x_{j}} [f_{m} \frac{∂f_{m}}{∂x_{i}} ]

= ∑_{m=1}^{M} [\frac{∂f_{m}}{∂x_{i}} \frac{∂f_{m}}{∂x_{j}} + f_{m} \frac{∂^{2}f_{m}}{∂x_{i}∂x_{j}} ]

≈ ∑_{m=1}^{M} \frac{∂f_{m}}{∂x_{i}} \frac{∂f_{m}}{∂x_{j}} = ∑_{m=1}^{M} J_{mi}J_{mj}  (2.20)

Here J_{ij} is the component of the Jacobian J_{f} in the ith row and jth column, and we have dropped the second order terms f_{m} (∂^{2}f_{m}/∂x_{i}∂x_{j}). Now the Hessian matrix can be written as

H_{ε}(x) = \frac{d^{2}}{dx^{2}} ε(x) = \frac{d}{dx}g_{ε}(x) ≈ J_{f}^{T} J_{f}  (2.21)

Based on Newton-Raphson’s method, the optimal solution x^{∗} that minimizes ε(x) can be found iteratively:

x_{n+1} = x_{n} − H_{n}^{−1}g_{n} ≈ x_{n} − (J_{n}^{T} J_{n})^{−1}J_{n}^{T} f_{n}  (2.22)

This is the same as pseudo-inverse method in Eq. (1.110) in Chapter 1. Example 2: Consider the following quadratic function:

q(x, y) = \frac{1}{2}[x_{1}, x_{2}] [ \begin{matrix}a & b/2 \\ b/2 & c \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = \frac{1}{2}(ax_{1}^{2} + bx_{1}x_{2} + cx_{2}^{2})

We have

g = [ \begin{matrix}ax_{1} + bx_{2}/2 \\ bx_{1}/2 + cx_{2} \end{matrix} ] , H = [ \begin{matrix}a & b/2 \\ b/2 & c \end{matrix} ] , det H = ac − b^{2}/4

We let a = 1, c = 2, and consider the following values of b:

• b = −2, λ_{1} = 2.618, λ_{2} = 0.382, det H = λ_{1}λ_{2} = 1, H > 0 is positive definite, f(0, 0) = 0 is the minimum.

• b = 0, λ_{1} = 2, λ_{2} = 1, det H = λ_{1}λ_{2} = 2, H > 0 is positive definite, f(0, 0) = 0 is the minimum.

• b = 2, λ_{1} = 2.618, λ_{2} = 0.382, det H = λ_{1}λ_{2} = 1, H > 0 is positive definite, f(0, 0) = 0 is the minimum.

• b = 4, λ_{1} = 3.562, λ_{2} = −0.562, det H = λ_{1}λ_{2} = −2, H is indefinite, f(0, 0) = 0 is a saddle point (minimum in one direction but maximum in another).

We can speed up the convergence by a bigger step size δ > 1. However, if δ is too big, the solution may be skipped and the iteration may not converge if it gets into an oscillation around the solution. Even worse, the iteration may become divergent. For such reasons, a smaller step size δ < 1 may be preferred sometimes.

In summary, Newton’s method approximates the function f(x) at an estimated solution x_{n} by a quadratic equation (the first three terms of the Taylor’s series) based on the gradient g_{n} and Hessian H of the function at x_{n}, and treat the vertex of the quadratic equation as the updated estimate x_{n+1}.

Newton’s method requires the Hessian matrix as well as the gradient to be available. Moreover, it is necessary calculate the inverse of the Hessian matrix in each iteration, which may be computationally expensive.

Example 3:

The Newton’s method is applied to solving the following non-linear equation system of N = 3 variables:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

with the exact solution (x_{1} = 0.5, x_{2} = 0, x_{3} = −0.5). These equations can be expressed in vector form as f(x) = 0 and solved as an optimization problem with the objective function J(x) = f^{T} (x)f(x). The iteration from an initial guess x_{0} = 0 is shown below.

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & ||f(x)|| \\ 0 & 0.000000, 0.000000, 0.000000 & 1.016120e + 01 \\ 1 & 0.500000, 0.500000, −0.500000 & 1.552502e + 02 \\ 2 & 0.499550, 0.250800, −0.493801 & 3.881300e + 01 \\ 3 & 0.500096, 0.126206, −0.496852 & 9.702208e + 00 \\ 4 & 0.500025, 0.063914, −0.498405 & 2.425198e + 00 \\ 5 & 0.500010, 0.032778, −0.499181 & 6.059054e − 01 \\ 6 & 0.500005, 0.017231, −0.499570 & 1.510777e − 01 \\ 7 & 0.500003, 0.009498, −0.499763 & 3.737330e − 02 \\ 8 & 0.500002, 0.005712, −0.499857 & 8.959365e − 03 \\ 9 & 0.500001, 0.003968, −0.499901 & 1.900145e − 03 \\ 10 & 0.500001, 0.003326, −0.499917 & 2.577603e − 04 \\ 11 & 0.500001, 0.003206, −0.499920 & 8.932714e − 06 \\ 12 & 0.500001, 0.003202, −0.499920 & 1.238536e − 08 \\ 13 & 0.500001, 0.003202, −0.499920 & 2.371437e − 14 \end{matrix}  (2.23)

We see that after 13 iterations the algorithm converges to the following approximated solution with accuracy of J(x) = ||f(x^{∗})||^{2} ≈ 10^{−28}:

x^{∗} = [ \begin{matrix}0.5000008539707297 \\ 0.0032017070323056 \\ −0.4999200212218281 \end{matrix} ]  (2.24)

2.1.3 Gradient Descent Method

Newton’s method discussed above is based on the gradient g_{f}(x) and Hessian H_{f}(x), representing the local variation and more global curvature of the function f(x) to be minimized. The method is not applicable if the Hessian H_{f} is not available, becaus either the analytical form of the function is not known or the cost for computing the inverse H_{f}^{−1} is too high. In such cases, we can use the gradient descent method based only on the gradient but no longer the Hessian.

We first consider the minimization of a single-variable function f(x). From any inital point x_{0}, we can move to a nearby point x along the opposite direction of the gradient, so that the function value will be smaller than that at the current position x_{0}:

x = x_{0} − δf′(x_{0}) = x_{0} + Δx_{0}, where Δx_{0} = −δf′(x_{0}), δ > 0 (2.25)

We see that no matter whether f′(x_{0}) is positive or negative, the function value f(x) approximated by the first two terms of its Taylor series is always reduced if the positive step size delta is small enough:

f(x) ≈ f(x_{0}) + f′(x_{0})Δx_{0} = f(x_{0}) − |f′(x_{0})|^{2}δ < f(x_{0})  (2.26)

This process can be carried out iteratively

x_{n+1} = x_{n} − δ_{n} f′(x_{n}),  n = 0, 1, 2, ⋯  (2.27)

until eventually reaching a point x^{∗} at which f′(x^{∗}) = 0 and no further progress can be made, i.e. a local minimum of the function is reached.

This simple method can be generalized to minimize a multi-variable objective function f(x) = f(x_{1}, ⋯, x_{N}) in N-D space. The derivative of the 1-D case is generalized to the gradient vector g(x) = df(x)/dx of function f(x), which is in the direction along which the function increases most rapidly with the steepest slope, perpendicular to the contour or iso-lines of the function f(x). The fastest way to reduce f(x) is to go down hill along the opposite direction of the gradient vector.

Specifically the gradient descent method (also called steepest descent or down hill method) carries out the iteration x = x_{0} + Δx, which is can be considered as the first two terms of the Taylor series of f(x) with Δx = −δg (δ > 0):

f(x) ≈ f(x_{0}) + g_{0}^{T} Δx = f(x_{0}) − δg_{0}^{T} g_{0} = f(x_{0}) − δ||g||^{2} < f(x_{0}) (2.28)

As the function value is always reduced by this iteration, it can be carried out repeatedly

x_{n+1} = x_{n} − δ_{n} g_{n} = (x_{n−1} − δ_{n−1} g_{n−1}) − δ_{n} g_{n} = ⋯ = x_{0} − ∑_{i=0}^{n} δ_{n} g_{i} (2.29)

untill eventually reaching a point x^{∗} at which g(x^{∗}) = 0 and f(x) is minimized.

Comparing Eq. (2.29) with Newton’s method where the search direction is d_{n} = −H_{n}^{−1}g_{n} in Eq. (2.16) based on both H_{n} and g_{n}, we see that here in gradient descent method the earch direction d_{n} = −g_{n} relies only on the local information contained in the gradient g without more global information contained in the second order Hessian H, i.e., the gradient method does not have as much information as Newton’s method and therefore may not be as efficient. For example, when the function is quadratic, as discussed before, Newton’s method can find the solution in a single step from any initial guess, but it takes the gradient descent method multiple steps to reach the solution, as it always follows the negative direction of the local gradient, which does not lead toward the solution directly in general. However, the gradient descent method is computationally less expensive as the Hessian matrix is not needed.

Example 1:

Consider a two-variable quadratic function in the following general form:

f(x) = x^{T} Ax = [x_{1} x_{2}] [ \begin{matrix}a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]

If

A = [ \begin{matrix}2 & 1 \\ 1 & 1 \end{matrix} ]

is a symmetric positive semidefinite matrix, i.e., a_{12} = a_{21} and A ≥ 0, then the function becomes

f(x_{1}, x_{2}) = \frac{1}{2}[x_{1} x_{2}] [ \begin{matrix}2 & 1 \\ 1 & 1 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = \frac{1}{2}(2x_{1}^{2} + 2x_{1}x_{2} + x_{2}^{2})

which has a minimum f(x_{1}, x_{2}) = 0 at x_{1} = x_{2} = 0.

We let the initial guess be x_{0} = [1, 2]^{T}, at which the gradient is g_{0} = [4, 3]^{T}. Now we compare the gradient method with Newton’s method, in terms of the search direction d and progress of the iterations:

• Newton’s method: Here we have

g = [ \begin{matrix}2x_{1} + x_{2} \\ x_{1} + x_{2} \end{matrix} ] , H = A = [ \begin{matrix}2 & 1 \\ 1 & 1 \end{matrix} ] , H^{−1} = [ \begin{matrix}1 & −1 \\ −1 & 2 \end{matrix} ]

and the search direction is d_{0} = −H^{−1}g_{0} = −[1, 2]^{T} (the red arrow in the figure). The iteration is:

x_{1} = x_{0} − H^{−1}g_{0} = [ \begin{matrix}1 \\ 2 \end{matrix} ] − [ \begin{matrix}1 & −1 \\ −1 & 2 \end{matrix} ] [ \begin{matrix}4 \\ 3 \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ]

which is the minimum of the function.

• The gradient descent method: the search direction is d_{0} = −g_{0} = −[4, 3]^{T}, perpendicular to the contour of the function. The first iteration is:

x_{1} = x_{0} − δg_{0} = [ \begin{matrix}1 \\ 2 \end{matrix} ] − δ [ \begin{matrix}4 \\ 3 \end{matrix} ] = [ \begin{matrix}1 − δ4 \\ 2 − δ3 \end{matrix} ]

We need to determine the step size δ (to be considered later) to find x_{1}, and then continue the iteration.

The figure below shows the search direction of the Newton’s method (red), which reaches the solution in a single step, based on not only the gradient g_{0}, the local information at the point x_{0}, but also the Hessian matrix H, the more global information of the quadratic function representing its curvature, in comparison to the gradient descent method (blue) based only on the local gradient g_{0}. Without global information H representing the elliptical shape of the contour line of the quadratic function, the gradient descent method requires an iteration, and is therefore less effective.

In the gradient descent method, we also need to determine a proper step size δ. If δ is too small, the iteration may converge too slowly, especially when f(x) reduces slowly toward its minimum. On the other hand, if δ is too large but f(x) has some rapid variations in the local region, the minimum of the function may be skipped and the iteration may not converge. We will consider how to find the optimal step size in the next section.

2.1.4 Line minimization

In general, in any iterative algorithm for minimizing a function f(x), including Newton’s method (Eq. (2.16)) and the gradient descent method (Eq. (2.29)), the variable x is updated iteratively x_{n+1} = x_{n} + δ_{n}d_{n} based on both the search direction d_{n} and the step size δ, which need to be determined in such a way that the function value f(x) is maximally reduced at each iteration step.

First, the search direction d_{n} needs to point away from the gradient g_{n} along which f(x_{n}) increases most rapidly. In other words, the angle between d_{n} and g_{n} should be greater than π/2:

cos^{−1} (\frac{d_{n}^{T} g_{n}}{||d_{n}|| ||g_{n}||}) > \frac{π}{2}  i.e.,  d_{n}^{T} g_{n} < 0  (2.30)

This condition is indeed satisfied in both the gradient descent method with d_{n} = −g_{n} and Newton’s method with d_{n} = −H_{n}^{−1}g_{n}:

d_{n}^{T} g_{n} = −g_{n}^{T} H_{n}^{−1}g_{n} < 0,  d_{n}^{T} g_{n} = −g_{n}^{T} g_{n} = −||g_{n}||^{2} < 0  (2.31)

Here H_{n} is assumed to be positive definite so that a minimum exists.

Next, the optimal step size δ_{n} needs to be such that the function value f(x_{n+1}) = f(x_{n} + δ_{n}d_{n}) is minimized along the search direction d_{n}. To do so, we set to zero the derivative of the function value with respect to δ_{n}, and get the directional derivative (Section A.18) along the direction of d_{n} based on the chain rule:

\frac{d}{dδ_{n}} f(x_{n+1}) = \frac{d}{dδ_{n}} f(x_{n}+δ_{n}d_{n}) = (\frac{d f(x_{n+1})}{dx})^{T} \frac{d(x_{n} + δ_{n}d_{n})}{dδ_{n}} = g_{n+1}^{T}d_{n} = 0

(2.32)

This result indicates that the gradient g_{n+1} = f′(x_{n+1}) at the next point x_{n+1} should be perpendicular to the search direction d_{n}. In other words, when traversing along d_{n}, we should stop at the point x_{n+1} = x_{n} +δd_{n} at which the gradient g_{n+1} is perpendicular to d_{n}, i.e., it has zero component along d_{n}, and the corresponding δ is the optimal step size.

To find the actual optimal step size δ, we need to solve a 1-D optimization problem while treating f(x_{n+1}) = f(x_{n} + δd_{n}) as a function of the step size δ, and approximate it by the first three terms of its Taylor series at δ = 0 (Maclaurin series of function f(δ)):

f(x_{n+1}) = f(x_{n} + δd_{n})
≈ [f(x_{n} + δd_{n})]_{δ=0} + δ [\frac{d}{dδ} f(x_{n} + δd_{n})]_{δ=0} + \frac{δ^{2}}{2} [\frac{d^{2}}{dδ^{2}} f(x_{n} + δd_{n})]_{δ=0} (2.33)

where

[f(x_{n} + δd_{n})]_{δ=0} = f(x_{n})  (2.34)
[\frac{d}{dδ} f(x_{n} + δd_{n})]_{δ=0} = [g(x_{n} + δd_{n})^{T} d_{n}]_{δ=0} = g_{n}^{T} d_{n}  (2.35)
[\frac{d^{2}}{dδ^{2}} f(x_{n} + δd_{n})]_{δ=0} = [\frac{d}{dδ} g(x_{n} + δd_{n})^{T} ]_{δ=0} \frac{d}{dδ} (x_{n} + δd_{n})
= [\frac{d}{dx}g(x) \frac{d}{dδ} (x_{n} + δd_{n})]_{δ=0}^{T} d_{n}
= (H_{n}d_{n})^{T} d_{n} = d_{n}^{T} H_{n}d_{n}  (2.36)

where H_{n} = H_{n}^{T} is the Hessian matrix of f(x) at x_{n}. Substituting these back into the Taylor series above we get:

f(x_{n} + δd_{n}) ≈ f(x_{n}) + δ g_{n}^{T} d_{n} + \frac{δ^{2}}{2} d_{n}^{T} H_{n} d_{n}  (2.37)

To find δ that minimizes f(x_{n} + δd_{n}), we set to zero its derivative with respect to δ:

\frac{d}{dδ} f(x_{n} + δd_{n}) ≈ \frac{d}{dδ} (f(x_{n}) + δg_{n}^{T} d_{n} + \frac{δ^{2}}{2} d_{n}^{T} H_{n} d_{n})
= g_{n}^{T} d_{n} + δ d_{n}^{T} H_{n} d_{n} = 0  (2.38)

and solve the resulting equation for δ to get the optimal step size based on both g_{n} and H_{n}:

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}}  (2.39)

Given this result, we can find the optimal step side for the Newton’s method and gradient descent method considered previously:

• Newton’s method:

The search direction is d_{n} = −H_{n}^{−1}g_{n}, and the optimal step size is

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}} = \frac{g_{n}^{T} (H_{n}^{−1}g_{n})}{(H_{n}^{−1}g_{n})^{T} H_{n}(H_{n}^{−1}g_{n})} = \frac{g_{n}^{T} (H_{n}^{−1}g_{n})}{(H_{n}^{−1}g_{n})^{T} g_{n}} = 1 (2.40)

This confirms that the iteration in Eq. (2.16) is indeed optimal:

x_{n+1} = x_{n} + δ_{n}d_{n} = x_{n} − δ_{n}H_{n}^{−1}g_{n} = x_{n} − H_{n}^{−1}g_{n}  (2.41)

• Gradient descent method:

The search direction is d_{n} = −g_{n}, and Eq. (2.32) becomes:

g_{n+1}^{T}d_{n} = −g_{n+1}^{T}g_{n} = 0, i.e., g_{n+1} ⊥ g_{n}  (2.42)

i.e., the search direction d_{n+1} = −g_{n+1} is always perpendicular to the previous one d_{n} = −g_{n}, i.e., the iteration follows a zigzag path composed of a sequence of segments from the initial guess to the final solution. The optimal step size is

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}} = \frac{g_{n}^{T} g_{n}}{g_{n}^{T} H_{n}g_{n}} = \frac{||g_{n}||^{2}}{g_{n}^{T} H_{n}g_{n}}  (2.43)

Given H_{n} as well as g_{n}, we can find the optimal step size δ_{n} with complexity O(N^{2}) for each iteration. However, as we assume the Hessian H_{n} is not available in the gradient descent method, the optimal step size above cannot be computed. We can instead approximate f′′(δ)|_{δ=0} in the third term of Eq. (2.33) at two nearby points at δ = 0 and δ = σ, where σ is a small value:

[\frac{d^{2}}{dδ^{2}} f(x + δd)]_{δ=0} = [\frac{d}{dδ} f′(x + δd)]_{δ=0} = lim_{σ→0} \frac{f′(x + σd) − f′(x)}{σ}
≈ \frac{g^{T} (x + σd) d − g^{T} (x) d}{σ} = \frac{g_{σ}^{T} d − g^{T} d}{σ}  (2.44)

where g = g(x) and g_{σ} = g(x + σd). This approximation can be used to replace d_{n}^{T} H_{n}d_{n} in Eq. (2.38) abve:

\frac{d}{dδ} f(x_{n} + δd_{n}) ≈ g_{n}^{T} d_{n} + \frac{δ}{σ} (g_{σn}^{T}d_{n} − g_{n}^{T} d_{n}) = 0  (2.45)

Solving for δ we get the estimated optimal step size:

δ_{n} = −\frac{σ g_{n}^{T} d_{n}}{g_{σn}^{T}d_{n} − g_{n}^{T} d_{n}} = −\frac{σ g_{n}^{T} d_{n}}{(g_{σn} − g_{n})^{T} d_{n}}  (2.46)

Specifically in the gradient descent method with d_{n} = −g_{n}, this optimal step size becomes:

δ_{n} = −\frac{σ g_{n}^{T} d_{n}}{(g_{σn} − g_{n})^{T} d_{n}} = −\frac{σ g_{n}^{T} g_{n}}{(g_{σn} − g_{n})^{T} g_{n}} = \frac{σ ||g_{n}||^{2}}{||g_{n}||^{2} − g_{σn}^{T}g_{n}}  (2.47)

and the iteration becomes

x_{n+1} = x_{n} − δ_{n}g_{n} = x_{n} + \frac{σ ||g_{n}||^{2}}{g_{σn}^{T}g_{n} − ||g_{n}||^{2}} g_{n}  (2.48)

Example 2:

The gradient descent method applied to solve the same three-variable equation system previously solved by Newton’s method:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

The step size δ_{n} is determined by the secant method with σ = 10^{−6}. The iteration from an initial guess x_{0} = 0 is shown below:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & ||f(x)|| \\ 0 & 0.0000, 0.0000, 0.0000 & 1.032500e + 02 \\ 10 & 0.4246, −0.0073, −0.5002 & 1.535939e − 01 \\ 20 & 0.5015, 0.0064, −0.4998 & 2.448241e − 05 \\ 30 & 0.5009, 0.0057, −0.4998 & 9.178209e − 06 \\ 40 & 0.5006, 0.0052, −0.4998 & 4.17587e − 06 \\ 50 & 0.5004, 0.0049, −0.4999 & 2.122594e − 06 \\ 60 & 0.5003, 0.0047, −0.4999 & 1.182466e − 06 \\ 100 & 0.5001, 0.0043, −0.4999 & 1.805871e − 07 \\ 150 & 0.5000, 0.0041, −0.4999 & 2.720744e − 08 \\ 200 & 0.5000, 0.0041, −0.4999 & 4.934027e − 09 \\ 250 & 0.5000, 0.0040, −0.4999 & 9.630085e − 10 \\ 300 & 0.5000, 0.0040, −0.4999 & 1.939747e − 10 \\ 350 & 0.5000, 0.0040, −0.4999 & 3.961646e − 11 \\ 400 & 0.5000, 0.0040, −0.4999 & 8.140393e − 12 \\ 450 & 0.5000, 0.0040, −0.4999 & 1.677968e − 12 \\ 500 & 0.5000, 0.0040, −0.4999 & 3.462695e − 13 \\ 550 & 0.5000, 0.0040, −0.4999 & 7.136628e − 14 \\ 600 & 0.5000, 0.0040, −0.4999 & 1.474205e − 14 \end{matrix}

We see that after the first 100 iterations the error is reduced to about J(x) = ||f(x^{∗})||^{2} ≈ 10^{−14}, and With 500 additional iterations the algorithm converges to the following approximated solution with accuracy of J(x) ≈ 10^{−28}:

x^{∗} = [ \begin{matrix}0.5000013623816102 \\ 0.0040027495837189 \\ −0.4999000311539049 \end{matrix} ]  (2.49)

Although the gradient descent method requires many more iterations than Newton’s method to converge, the computational cost in each iteration is much reduced, as no more matrix inversion is needed.

When it is difficult or too computationally costly to find the optimal step size along the search direction, some suboptimal step size may be acceptable, such as in the quasi-Newton methods (Section 2.1.5) for minimization problems. In this case, although the step size δ is no longer required to be such that the function value at the new position f(x_{n+1}) = f(x_{n} + δd_{n}) is minimized along the search direction d_{n}, the step size δ still has to satisfy the following Wolfe conditions:

• Sufficient decrease (Armijo rule):

f(x_{n+1}) = f(x_{n} + δd_{n}) ≤ f(x_{n}) + c_{1} δd^{T} g_{n}  (2.50)

• Curvature condition:

g_{n+1}^{T}d_{n} ≥ c_{2} g_{n}^{T} d_{n}  (2.51)

As g_{n}^{T} d_{n} < 0, this condition can also be written in the following alternative form:

|g_{n+1}^{T}d_{n}| < |c_{2} g_{n}^{T} d_{n}|  (2.52)

Here the two constants c_{1} and c_{2} above satisfy 0 < c_{1} < c_{2} < 1.

In general, these conditions are motivated by the desired effect that after each iterative step, the function should have a shallower slope along d_{n}, as well as a lower value, so that eventually the solution can be approached where f(x) is minimum and the gradient is zero.

Specifically, to understand the first condition above, we represent the function to be minimized as a single-variable function of the step size ϕ(δ) = f(x_{n} +δd_{n}), and its tangent line at the point δ = 0 as a linear function L_{0}(δ) = a + bδ, where the intercept a can be found as a = L_{0}(0) = ϕ(δ)|_{δ=0} = f(x_{n}), and the slope b can be found as the derivative of ϕ(δ) = f(x_{n} + δd_{n}) at δ = 0:

b = \frac{d}{dδ} f(x_{n} + δd_{n})|_{δ=0} = g_{n}^{T} d_{n} < 0  (2.53)

which is required to be negative for the function value to be reduced, f(x_{n} + δd_{n}) < f(x_{n}). Now the function of the tangent line can be written as

L_{0}(δ) = a + bδ = f(x_{n}) + g_{n}^{T} d_{n}δ  (2.54)

Comparing this with a constant line L_{1}(δ) = f(x_{n}) of slope zero, we see that any straight line between L_{0}(δ) and L_{1}(δ) can be described by L(δ) = f(x_{n}) + c_{1} g_{n}^{T} d_{n}δ with 0 < c_{1} < 1, with a slope 0 < c_{1}g_{n}^{T} d_{n} < g_{n}^{T} d_{n}. The Armijo rule is to find any δ > 0 that satisfies

f(x_{n+1}) = f(x_{n} + δd_{n}) ≤ f(x_{n}) + c_{1} g_{n}^{T} d_{n}δ < f(x_{n})  (2.55)

We see that the value of f(x) is guaranteed to be reduced.

The second condition requires that at the new position x_{n+1} the slope of the gradient g_{n+1} along the search direction d_{n} be sufficiently reduced to be less than a specified value (determined by c_{2}), in comparison to the slope at the old position x_{n}.

The reason why g_{n+1} ⊥ g_{n} can also be explained geometrically. As shown in the figure the gradient vectors at various points along the direction of −g_{n} are shown by the blue arrows, and their projections onto the direction represent the slopes of the function f(δ_{n}) = f(x_{n} − δ_{n}g_{n}). Obviously at x_{n+1} where f(δ_{n}) reaches its minimum, its slope is zero. In other words, the projection of the gradient g_{n+1} onto the direction of −g_{n} is zero, i.e., g_{n+1} ⊥ g_{n} or g_{n+1}^{T}g_{n} = 0.

The gradient descent method gradually approaches a solution x of an N-D minimization problem by moving from the initial guess x_{0} along a zigzag path composed of a set of segments with any two consecutive segments perpendicular to each other. The number of steps depends greatly on the initial guess. As illustrated in the example below in an N=2 dimensional case, the best possible case is that the solution happens to be on the gradient direction of the initial guess, which could be reached in a single step, while the worst possible case is that the gradient direction of the initial guess happens to be 45 degrees off from the gradient direction of the optimal case, and it takes many zigzag steps to go around the optimal path to reach the solution. Many of the steps are in the same direction as some of the previous steps.

To improve the performance of the gradient descent method we can include in the iteration a momentum term representing the search direction previously traversed:

x_{n+1} = x_{n} − δ_{n}g_{n} + α_{n}(x_{n} − x_{n−1})  (2.56)

Now two consecutive search directions are no longer perpendicular to each other and the resulting search path is smoother than the zigzag path without the momentum term. The parameter α_{m} controls how much momentum is to be added.

Obviously it is most desirable not to repeat any of the previous directions traveled so that the solution can be reached in N steps, each in a unique direction in the N-D space. In other words, the subsequent steps are independent of each other, never interfering with the results achieved in the previous steps. Such a method will be discussed in the next section.

Example 3:

The Rosenbrock function

f(x_{1}, x_{2}) = (a − x_{1})^{2} + b(x_{2} − x_{1}^{2})^{2} (e.g., a = 1, b = 100)

is a two-variable non-convex function with a global minimum f(1, 1) = 0 at the point (1, 1), which is inside a long parabolic shaped valley as shown in the figure below. As the slope along the valley is very shallow, it is difficult for an algorithm to converge quickly to the minimum. For this reason, the Rosenbrock function is often used to test various minimization algorithms.

The figure below shows the search path of the gradient descent method based on the optimal step size given in Eq. (2.43). We see that the search path is composed a long sequence of 90 degree turns between consecutive segments.

When the Newton’s method is applied to this minimization problem, it takes only four iterations for the algorithm to converge to the minimum, as shown in the figure below:

2.1.5 Quasi-Newton Methods

As we have seen above, Newton’s methods can be used to solve both nonlinear systems to find roots of a set of simultaneous equations, and optimization problems to minimize a scalar-valued objective function based on the iterations of the same form. Specifically,

• Solving equations f(x) = 0:

x_{n+1} = x_{n} − J(x_{n})^{−1}f(x_{n}) = x_{n} − J_{n}^{−1}f_{n}  (2.57)

where J_{n} = J(x_{n}) is the Jacobian matrix of f(x_{n}),

• Minimizing f(x):

x_{n+1} = x_{n} − H(x_{n})^{−1}g(x_{n}) = x_{n} − H_{n}^{−1}g_{n}  (2.58)

where g_{n} = g(x_{n}) and H_{n} = H(x_{n}) are the gradient vector and Hessian matrix of f(x) at x_{n}, respectively.

If the Jacobian J(x) or the Hessian matrix H(x) is not available, or if it is too computationally costly to calculate its inverse (with complexity O(N^{3})), the quasi-Newton methods can be used to approximate the Hessian matrix or its inverse based only on the first order derivative, the gradient g of f(x) (with complexity O(N^{2})), similar to Broyden’s method previously considered (Section 1.5).

In the following, we consider the minimization of a function f(x). Its Taylor expansion around point x_{n+1} is

f(x) = f(x_{n+1})+(x−x_{n+1})^{T} g_{n+1}+\frac{1}{2}(x−x_{n+1})^{T} H_{n+1}(x−x_{n+1})+O(||x−x_{n+1}||^{3})

(2.59)

Taking derivetive with respect to x, we get

\frac{d}{dx}f(x) = g(x) = g_{n+1} + H_{n+1}(x − x_{n+1}) + O(||x − x_{n+1}||^{2})  (2.60)

Evaluating at x = x_{n}, we have g(x_{n}) = g_{n}, and the above can be written as

g_{n+1} − g_{n} = H_{n+1}(x_{n+1} − x_{n}) + O(||x_{n+1} − x_{n}||^{2})
= B_{n+1}(x_{n+1} − x_{n})  (2.61)

where matrix B_{n} is the secant approximation of the Hessian matrix H_{n}, and the last eqality is called the secant equation. For convenience, we further define:

s_{n} = x_{n+1} − x_{n}, y_{n} = g_{n+1} − g_{n}  (2.62)

so that the equation above can be written as

B_{n+1}s_{n} = y_{n}, or B_{n+1}^{−1}y_{n} = s_{n}  (2.63)

This is the quasi-Newton equation, which is the secant condition that must be satisfied by matrix B_{n}, or its inverse B_{n}^{−1}, in any of the quasi-Newton algorithms, all taking the follow general steps:

1. Initialize x_{0} and B_{0}, set n = 0;

2. Compute gradient g_{n} and the search direction d_{n} = −B_{n}^{−1}g_{n};

3. Get x_{n+1} = x_{n} + δd_{n} with step size δ satisfying the Wolfe conditions

4. update B_{n+1} = B_{n} + ΔB_{n} or B_{n+1}^{−1} = B_{n}^{−1} + ΔB_{n}^{−1}, that satisfies the quasi-Newton equation;

5. If termination condition is not satisfied, n = n + 1, go back to second step.

For this iteration to converge to a local minimum of f(x), B_{n} must be positive definite matrix, same as the Hessian matrix H it approximates. Specially, if B_{n} = I, then d_{n} = −g_{n}, the algorithm becomes the gradient descent method; also, if B_{n} = H_{n} is the Hessian matrix, then the algorithm becomes the Newton’s method.

In a quasi-Newton method, we can choose to update either B_{n} or its inverse B_{n}^{−1} based on one of the two forms of the quasi-Newton equation. We note that there is a dual relationship between the two ways to update, i.e., by swapping y_{n} and s_{m}, an update formula for B_{n} can be directly applied to its inverse and vice versa.

• The Symmetric Rank 1 (SR1) Algorithm Here matrix B_{n} is updated by an additional term:

B_{n+1} = B_{n} + uu^{T}  (2.64)

where u a vector and uu^{T} is a matrix of rank 1. Imposing the secant condition, we get

B_{n+1}s_{n} = B_{n}s_{n} + uu^{T} s_{n} = y_{n}, i.e., u(u^{T} s_{n}) = y_{n} − B_{n}s_{n} (2.65)

This equation indicats u is in the same direction as w = y_{n} − B_{n}s_{n}, and can therefore be written as u = cw. Substituting this back we get

uu^{T} s_{n} = c^{2}ww^{T} s_{n} = w  (2.66)

Solving this we get c = 1/(w^{T} s_{n})^{1/2},

u = cw = \frac{y_{n} − B_{n}s_{n}}{(w^{T} s_{n})^{1/2}} = \frac{y_{n} − B_{n}s_{n}}{((y_{n} − B_{n}s_{n})^{T} s_{n})^{1/2}}  (2.67)

and the iteration of B_{n} becomes

B_{n+1} = B_{n} + uu^{T} = B_{n} + \frac{(y_{n} − B_{n}s_{n})(y_{n} − B_{n}s_{n})^{T}}{(y_{n} − B_{n}s_{n})^{T} s_{n}}  (2.68)

Applying the Sherman-Morrison formula (Section A.6) we further get the inverse B_{n+1}:

B_{n+1}^{−1} = (B_{n} + uu^{T} )^{−1} = B_{n}^{−1} − \frac{B_{n}^{−1}uu^{T} B_{n}^{−1}}{1 + u^{T} B_{n}^{−1}u}
= B_{n}^{−1} + \frac{(s_{n} − B_{n}^{−1}y_{n})(s_{n} − B_{n}^{−1}y_{n})^{T}}{(s_{n} − B_{n}^{−1}y_{n})^{T} y_{n}}  (2.69)

We note that this equation is dual to the previous one, and it can also be obtained based on the duality between the secant conditions for B_{n} and B_{n}^{−1}. Specifically, same as above, we impose the secant condition for the inverse B_{n+1}^{−1} of the following form

B_{n+1}^{−1} = B_{n}^{−1} + uu^{T}  (2.70)

and get

B_{n+1}^{−1}y_{n} = B_{n}^{−1}y_{n} + uu^{T} y_{n} = s_{n},  (2.71)

Then by repeating the same process above, we get the same update formula for B_{n}^{−1}.

This is the formula for directly updating B_{n}^{−1}. For the iteration to converge to a local minimum, matrix B_{n+1}^{−1} needs to be positive definite as well as B_{n}^{−1}, we therefore require

w^{T} y_{n} = (s_{n} − B_{n}^{−1}y_{n})^{T} y_{n} = s_{n}^{T} y_{n} − y_{n}^{T} B_{n}^{−1}y_{n} > 0  (2.72)

It may be difficult to maintain this requirement through out the iteration. This problem can be avoided in the following DFP and BFGS methods.

• The BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm

This is a rank 2 algorithm in which matrix B_{n} is updated by two rank-1 terms:

B_{n+1} = B_{n} + αuu^{T} + βvv^{T}  (2.73)

Imposing the secant condition for B_{n}, we get

B_{n+1}s_{n} = (B_{n} + αuu^{T} + βvv^{T})s_{n} = B_{n}s_{n} + u(αu^{T} s_{n}) + v(βv^{T} s_{n}) = y_{n}

(2.74)

or

u(αu^{T} s_{n}) + v(βv^{T} s_{n}) = y_{n} − B_{n}s_{n}  (2.75)

which can be satisfied if we let

u = y_{n}, α = \frac{1}{s_{n}^{T} y_{n}} , v = B_{n}s_{n}, β = −\frac{1}{v^{T} s_{n}} = −\frac{1}{s_{n}^{T} B_{n}s_{n}} (2.76)

Substituting these into B_{n+1} = B_{n} + αuu^{T} + βvv^{T} we get

B_{n+1} = B_{n} + αuu^{T} + βvv^{T} = B_{n} + \frac{y_{n}y_{n}^{T}}{y_{n}^{T} s_{n}} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}}  (2.77)

Given this update formula for B_{n}, we can further find its inverse B_{n}^{−1} and then the search direction d_{n} = −B_{n}^{−1}g_{n}.

Alternatively, given B_{n}, we can further derive an update formula directly for the inverse matrix B_{n}^{−1}, so that the search direction d_{n} can be obtained without carrying out the inversion computation. Specifically, we first define U = [u_{1} u_{2}] and V = [v_{1} v_{2}], where

u_{1} = v_{1} = \frac{y_{n}}{(s_{n}^{T} y_{n})^{1/2}} , u_{2} = −v_{2} = \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}}  (2.78)

so that the expression above can be written as:

B_{n+1} = B_{n} + u_{1}v_{1}^{T} + u_{2}v_{2}^{T} = (B_{n} + UV^{T})^{−1}  (2.79)

2.1 Unconstrained Optimization 51

and then apply the Sherman-Morrison formula to get:

B_{n+1}^{−1} = (B_{n} + UV^{T})^{−1} = B_{n}^{−1} − B_{n}^{−1}U(I + V^{T} B_{n}^{−1}U)^{−1}V^{T} B_{n}^{−1}
= B_{n}^{−1} − B_{n}^{−1}UC^{−1}V^{T} B_{n}^{−1}
= B_{n}^{−1} − B_{n}^{−1}[u_{1} u_{2}]C^{−1}(B_{n}^{−1}[v_{1} v_{2}])^{T}  (2.80)

where we have defined

C = [ \begin{matrix}c_{11} & c_{12} \\ c_{21} & c_{22} \end{matrix} ] = I + V^{T} B_{n}^{−1}U = I + [v_{1} v_{2}]^{T} B_{n}^{−1}[u_{1} u_{2}] (2.81)

with

c_{11} = 1 + v_{1}^{T} B_{n}^{−1}u_{1} = 1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}}  (2.82)
c_{22} = 1 + v_{2}B_{n}^{−1}u_{2} = 1 − \frac{s_{n}^{T} B_{n}B_{n}^{−1}B_{n}s_{n}}{s_{n}^{T} B_{n}s_{n}} = 0  (2.83)
c_{12} = v_{1}^{T} B_{n}^{−1}u_{2} = \frac{y_{n}^{T} B_{n}^{−1}B_{n}s_{n}}{(s_{n}^{T} y_{n})^{1/2}(s_{n}^{T} B_{n}s_{n})^{1/2}} = \frac{(s_{n}^{T} y_{n})^{1/2}}{(s_{n}^{T} B_{n}s_{n})^{1/2}}  (2.84)
c_{21} = v_{2}^{T} B_{n}^{−1}u_{1} = −c_{12}  (2.85)
C = [ \begin{matrix}c_{11} & c_{12} \\ −c_{12} & 0 \end{matrix} ] , C^{−1} = [ \begin{matrix}0 & −1/c_{12} \\ 1/c_{12} & c_{11}/c_{12}^{2} \end{matrix} ] (2.86)

Substituting this C^{−1} into the expression for B_{n+1}^{−1}, we get:

B_{n+1}^{−1} = B_{n}^{−1} − [B_{n}^{−1}u_{1} B_{n}^{−1}u_{2}] [ \begin{matrix}0 & −1/c_{12} \\ 1/c_{12} & c_{11}/c_{12}^{2} \end{matrix} ] [B_{n}^{−1}v_{1} B_{n}^{−1}v_{2}]^{T}
= B_{n}^{−1} − \frac{1}{c_{12}} [B_{n}^{−1}u_{2}v_{1}^{T} B_{n}^{−1} − B_{n}^{−1}u_{1}v_{2}^{T} B_{n}^{−1}] − \frac{c_{11}}{c_{12}^{2}} B_{n}^{−1}u_{2}v_{2}^{T} B_{n}^{−1} (2.87)

where

\frac{1}{c_{12}} B_{n}^{−1}u_{2}v_{1}^{T} B_{n}^{−1} = \frac{(s_{n}^{T} B_{n}s_{n})^{1/2}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} \frac{y_{n}^{T}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} = \frac{s_{n}y_{n}^{T} B_{n}^{−1}}{s_{n}^{T} y_{n}}

(2.88)

−\frac{1}{c_{12}} B_{n}^{−1}u_{1}v_{2}^{T} B_{n}^{−1} = \frac{(s_{n}^{T} B_{n}s_{n})^{1/2}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} \frac{y_{n}}{(s_{n}^{T} y_{n})^{1/2}} \frac{s_{n}^{T} B_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} B_{n}^{−1} = \frac{B_{n}^{−1}y_{n}s_{n}^{T}}{s_{n}^{T} y_{n}}

(2.89)

\frac{c_{11}}{c_{12}^{2}} B_{n}^{−1}u_{2}v_{2}^{T} B_{n}^{−1} = − (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}^{T} B_{n}s_{n}}{s_{n}^{T} y_{n}} B_{n}^{−1} \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} \frac{s_{n}^{T} B_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} B_{n}^{−1}
= − (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}}  (2.90)

Substituting these three terms back into Eq. (2.87), we get the update formula for B_{n}^{−1}:

B_{n+1}^{−1} = B_{n}^{−1} − \frac{B_{n}^{−1}y_{n}s_{n}^{T} + s_{n}y_{n}^{T} B_{n}^{−1}}{s_{n}^{T} y_{n}} + (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} (2.91)

In summary, B_{n+1}^{−1} can be found by either Eq. (2.77) for B_{n+1} followed by an additional inversion operation, or Eq. (2.91) directly for B_{n+1}^{−1}. The results obtained by these methods are equivalent.

• The DFP (Davidon-Fletcher-Powell) Algorithm

Same as the BFGS algorithm, the DFP algorithm is also a rank 2 algorithm, where instead of B_{n}, the inverse matrix B_{n}^{−1} is updated by two rank-1 terms:

B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T}  (2.92)

where α and β are real scalars, u and v are vectors. Imposing the secant condition for the inverse of B_{n}, we get

B_{n+1}^{−1}y_{n} = (B_{n}^{−1}+αuu^{T} +βvv^{T})y_{n} = B_{n}^{−1}y_{n}+u(αu^{T} y_{n})+v(βv^{T} y_{n}) = s_{n}

(2.93)

or

u(αu^{T} y_{n}) + v(βv^{T} y_{n}) = s_{n} − B_{n}^{−1}y_{n}  (2.94)

which can be satisfied if we let

u = s_{n}, α = \frac{1}{s^{T} y_{n}} , v = B_{n}^{−1}y_{n}, β = −\frac{1}{v^{T} y_{n}} = −\frac{1}{y_{n}^{T} B_{n}^{−1}y_{n}}

(2.95)

Substituting these into B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T} we get

B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T} = B_{n}^{−1} + \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} − \frac{B_{n}^{−1}y_{n}y_{n}^{T} B_{n}^{−1}}{y_{n}^{T} B_{n}^{−1}y_{n}} (2.96)

Following the same procedure used for the BFGS method, we can also obtain the update formular of matrix B_{n} for the DFP method:

B_{n+1} = B_{n} − \frac{B_{n}s_{n}y_{n}^{T} + y_{n}s_{n}^{T} B_{n}}{y_{n}^{T} s_{n}} + (1 + \frac{s_{n}^{T} B_{n}s_{n}}{y_{n}^{T} s_{n}} ) \frac{y_{n}y_{n}^{T}}{y_{n}^{T} s_{n}}  (2.97)

We note that Eqs. (2.96) and (2.77) form a duality pair, and Eqs. (2.97) and (2.91) form another duality pair. In other words, the BFGS and FDP methods are dual of each other.

In both the DFP and BFGS methods, matrix B_{n}^{−1}, as well as B_{n}, must be positive definite, i.e., z^{T} B_{n}z > 0 must hold for any z ≠ 0. We now prove that this requirement is satisfied if the curvature condition of the Wolfe conditions is satisfied:

g_{n+1}^{T}d_{n} ≥ c_{2} g_{n}^{T} d_{n} i.e. (g_{n+1} − c_{2} g_{n})^{T} d_{n} ≥ 0  (2.98)

Replacing the search direction d_{n} by δd_{n} = x_{n+1} − x_{n} = s_{n}, we get:

(g_{n+1} − c_{2}g_{n})^{T} s_{n} = g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n} ≥ 0  (2.99)

We also have

y_{n}^{T} s_{n} = (g_{n+1} − g_{n})^{T} s_{n} = g_{n+1}^{T}s_{n} − g_{n}^{T} s_{n}  (2.100)

Substracting the first equation from the second, we get

y_{n}^{T} s_{n} − (g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n}) = (c_{2} − 1)g_{n}^{T} s_{n} > 0  (2.101)

The last inequality is due to the fact thatg_{n}^{T} s_{n} < 0 and c_{2} < 1. We therefore have

y_{n}^{T} s_{n} ≥ g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n} ≥ 0  (2.102)

Given y_{n}^{T} s_{n} > 0, we can further prove by induction that both B_{n+1} and B_{n+1}^{−1} based repectively on the update formulae in Eqs. (2.77) and (2.96) are positive definite. We first assume B_{0} and B_{n} are both positive definite, and express B_{n} in terms of its Cholesky decomposition, B_{n} = LL^{T}. We further define a = L^{T} z, b = L^{T} s, and get

a^{T} a = z^{T} B_{n}z, b^{T} b = s^{T} B_{n}s, a^{T} b = z^{T} B_{n}s  (2.103)

Now we can show that the sum of the first and third terms of Eq. (2.91) is positive definite:

z^{T} [B_{n} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}} ] z = z^{T} B_{n}z − \frac{(z^{T} B_{n}s_{n})^{2}}{s_{n}^{T} B_{n}s_{n}} = a^{T} a − \frac{(a^{T} b)^{2}}{b^{T} b} ≥ 0 (2.104)

The last step is due to the Cauchy-Schwarz inequality. Also, as s_{n}^{T} y_{n} ≥ 0, the second term of Eq. (2.77) is positive definite

z^{T} (\frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} ) z ≥ 0  (2.105)

Combining these two results, we get

z^{T} B_{n+1}z = z^{T} (B_{n} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}} ) z + z^{T} (\frac{y_{n}y_{n}^{T}}{s_{n}^{T} y_{n}} ) z ≥ 0  (2.106)

i.e., B_{n+1} based on the update formular Eq. (2.77) is positive definite. Following the same steps, we can also show that B_{n+1}^{−1} based on Eq. (2.96) is positive definite.

Examples:

The figures below shows the search path of the DFP and BPGS methods when applied to find the minimum of the Rosenbrock function.

2.1.6 Conjugate gradient method

The gradient descent method can be used to solve the minimization problem when the Hessian matrix of the objective function is not available. However, this method may be inefficient if it gets into a zigzag search pattern and repeat the same search directions many times. This problem can be avoided by the conjugate gradient (CG) method. If the objective function is quadratic, the CG method converges to the solution in N iterations without repeating any of the directions previously traversed. If the objective function is not quadratic, the CG method can still significantly improve the performance in comparison to the gradient descent method.

Again consider the approximation of the function f(x) to be minimized by the first three terms of its Taylor series:

f(x) = f(x_{0} + δx) ≈ f(x_{0}) + g_{0}^{T} δx + \frac{1}{2}δx^{T} H_{0}δx  (2.107)

If the Hessian matrix is positive definite, then f(x) has a minimum. If function f(x) is quadratic, the approximation above becomes exact and the function can be written as

f(x) = \frac{1}{2}x^{T} Ax − b^{T} x + c  (2.108)

where A = H is the symmetric Hessian matrix, and its gradient and Hessian can be written as the following respectively:

g(x) = \frac{d}{dx}f(x) = \frac{d}{dx} (\frac{1}{2}x^{T} Ax − b^{T} x + c) = Ax − b  (2.109)

and

H(x) = \frac{d^{2}}{dx^{2}} f(x) = \frac{d}{dx}g = \frac{d}{dx}(Ax − b) = A  (2.110)

Solving g(x) = Ax − b = 0, we get the solution x^{∗} = A^{−1}b, at which the function is minimized to

f(x^{∗}) = \frac{1}{2}(A^{−1}b)^{T} A(A^{−1}b) − b^{T} (A^{−1}b) + c = −\frac{1}{2}b^{T} A^{−1}b + c (2.111)

At the solution x^{∗} that minimizes f(x), its gradient is

g(x^{∗}) = Ax^{∗} − b = 0  (2.112)

We also see that the minimization of the quadratic function f(x) is equivalent to solving a linear equation Ax = b with a symmetric positive definite coefficient matrix A. The CG method considered here can therefore be used for solving both problems.

Conjugate basis vectors

We first review the concept of conjugate vectors (Section A.2) which is of essential importance in the CG method. Two vectors u and v are mutually conjugate or A-conjugate to each other with respect to a symmetric matrix A = A^{T}, if they satisfy:

u^{T} Av = (Av)^{T} u = v^{T} Au = 0  (2.113)

Specially, if A = I, the two conjugate vectors become orthogonal to each other, i.e., v^{T} u = 0.

Similar to a set of N orthogonal vectors that can be used as the basis spanning an N-D space, a set of N mutually conjugate vectors {d_{0}, ⋯, d_{N−1}} satisfying d_{i}^{T} Ad_{j} = 0 (i ≠ j) can also be used as a basis to span the N-D space. Any vector in the space can be expressed as a linear combination of these basis vectors.

Also we note that any set of N independent vectors can be converted by the Gram-Schmidt process (Section A.2) to a set of N basis vectors that are either orthogonal or A-conjugate to each other.

Example:

Given two independent basis vectors of the 2-D space, and a positive-definite matrix:

v_{1} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , v_{2} = [ \begin{matrix}0 \\ 1 \end{matrix} ] , A = [ \begin{matrix}3 & 1 \\ 1 & 2 \end{matrix} ]

we can construct two A-orthogonal basis vectors by the Gram-Schmidt method:

u_{1} = v_{1} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , u_{2} = v_{2} − \frac{u_{1}^{T} Av_{2}}{u_{1}^{T} Au_{1}} u_{1} = [ \begin{matrix}−1/3 \\ 1 \end{matrix} ]

The projections of a vector x = [2, 3]^{T} onto v_{1} and v_{2} are:

p_{v_{1}}(x) = \frac{v_{1}^{T} x}{v_{1}^{T} v_{1}} v_{1} = 2 [ \begin{matrix}1 \\ 0 \end{matrix} ] = [ \begin{matrix}2 \\ 0 \end{matrix} ] , p_{v_{2}}(x) = \frac{v_{2}^{T} x}{v_{2}^{T} v_{2}} v_{2} = 3 [ \begin{matrix}0 \\ 1 \end{matrix} ] = [ \begin{matrix}0 \\ 3 \end{matrix} ]

The A-projections of the same vector x onto u_{1} and u_{2} are:

p_{u_{1}}(x) = \frac{u_{1}^{T} Ax}{u_{1}^{T} Au_{1}} u_{1} = 3 [ \begin{matrix}1 \\ 0 \end{matrix} ] = [ \begin{matrix}3 \\ 0 \end{matrix} ] , p_{u_{2}}(x) = \frac{u_{2}^{T} Ax}{u_{2}^{T} Au_{2}} u_{2} = 3 [ \begin{matrix}−1/3 \\ 1 \end{matrix} ] = [ \begin{matrix}−1 \\ 3 \end{matrix} ]

The original vector x can be represented in either of the two bases:

x = [ \begin{matrix}2 \\ 3 \end{matrix} ] = 2 [ \begin{matrix}1 \\ 0 \end{matrix} ] + 3 [ \begin{matrix}0 \\ 1 \end{matrix} ] = 2v_{1} + 3v_{2} = p_{v_{1}}(x) + p_{v_{2}}(x)
= 3 [ \begin{matrix}1 \\ 0 \end{matrix} ] + 3 [ \begin{matrix}−1/3 \\ 1 \end{matrix} ] = 3u_{1} + 3u_{2} = p_{u_{1}}(x) + p_{u_{2}}(x)

Search along a conjugate basis

Similar to the gradient descent method, which iteratively improves the estimated solution by following a sequence of orthogonal search directions {d_{0}, ⋯, d_{n}, ⋯ } with d_{n} = −g_{n} satisfying g_{n}^{T} g_{n+1} = 0, the CG method also follows a sequence of search directions {d_{0}, ⋯, d_{N−1}} A-orthogonal to each other, i.e., d_{i}^{T} Ad_{j} = 0 (i ≠ j):

x_{n+1} = x_{n} + δ_{n}d_{n} = ⋯ = x_{0} + ∑_{i=0}^{n} δ_{i}d_{i}  (2.114)

We define the error at the nth step as e_{n} = x_{n} − x^{∗}. Then subtracting x^{∗} from both sides, we get the iteration in terms of the errors:

e_{n+1} = e_{n} + δ_{n}d_{n} = ⋯ = e_{0} + ∑_{i=0}^{n} δ_{i}d_{i}  (2.115)

Due to g = Ax − b in Eq. (2.112), we can find the gradient at the nth step x_{n} as

g_{n} = Ax_{n} − b = A(x^{∗} + e_{n}) − b = Ax^{∗} − b + Ae_{n} = Ae_{n}  (2.116)

As the gradient at the solution is zero g(x^{∗}) = 0, we can consider the gradient g_{n} at x_{n} as the residual of the nth iteration, and ε = ||g_{n}||^{2} an error measurement representing how close x_{n} is to the true solution x^{∗}.

The optimal step size given in Eq. (2.39) can now be written as

δ_{i} = −\frac{d_{i}^{T} g_{i}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} Ae_{i}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} A (e_{0} + ∑_{j=0}^{i−1} δ_{j}d_{n})}{d_{i}^{T} Ad_{i}}
= −\frac{d_{i}^{T} Ae_{0} + ∑_{j=0}^{n−1} δ_{j}d_{i}^{T} Ad_{j}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}}  (2.117)

The last equality is due to the fact that d_{i} and d_{j} are A-orthogonal, d_{i}^{T} Ad_{j} = 0. Substituting this into Eq. (2.115) we get

e_{n+1} = e_{n} + δ_{n}d_{n} = e_{n} − (\frac{d_{n}^{T} Ae_{0}}{d_{n}^{T} Ad_{n}} ) d_{n}  (2.118)

On the other hand, we can represent the error e_{0} = x_{0} − x^{∗} associated with the initial guess x_{0} as a linear combination of the A-orthogonal search vectors {d_{0}, ⋯, d_{N−1}} as N basis vectors that span the N-D vector space:

e_{0} = ∑_{i=0}^{N−1} c_{i}d_{i} = ∑_{i=0}^{N−1} p_{d_{i}}(e_{0}) = ∑_{i=0}^{N−1} (\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.119)

where p_{d_{i}}(e_{0}) is the A-projection of e_{0} onto the ith basis vector d_{i}:

p_{d_{i}}(e_{0}) = c_{i}d_{i} = (\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.120)

Note that the coefficient c_{i} happens to be the negative optimal step size in Eq. (2.117):

c_{i} = \frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} = −δ_{i}  (2.121)

Now the expression of e_{n+1} in Eq. (2.115) can be written as

e_{n+1} = e_{0}+∑_{i=0}^{n} δ_{i}d_{i} = ∑_{i=0}^{N−1} c_{i}d_{i}−∑_{i=0}^{n} c_{i}d_{i} = ∑_{i=n+1}^{N−1} c_{i}d_{i} = ∑_{i=n+1}^{N−1} p_{d_{i}}(e_{0}) (2.122)

We see that in each iteration, the number of terms in the summation for e_{0} is reduced by one, i.e., the nth component p_{d_{n}}(e_{0}) of e_{0} along the direction of d_{n} is completely eliminated. After N such iterations, the error is reduced from e_{0} to e_{N} = 0, and the true solution is obtained x_{N} = x^{∗} + e_{N} = x^{∗}.

Pre-multiplying d_{k}^{T} A (k ≤ n) on both sides of the equation above, we get

d_{k}^{T} Ae_{n+1} = ∑_{i=n+1}^{N−1} c_{i}d_{k}^{T} Ad_{j} = 0  (2.123)

We see that after n + 1 iterations the remaining error e_{n+1} is A-orthogonal to all previous directions d_{0}, ⋯, d_{n}.

Due to Eq. (2.116), the equation above can also be written as

d_{k}^{T} Ae_{n+1} = d_{k}^{T} g_{n+1} = 0  (2.124)

i.e., the gradient g_{n+1} is orthogonal to all previous search directions.

In the figure below, the conjugate gradient method is compared with the gradient descent method for the case of N = 2. We see that the first search direction is the same −g_{0} for both methods. However, the next search direction d_{1} is Aorthogonal to d_{0}, same as the next error e_{1}, different from the search direction −g_{1} in gradient descent method. The conjugate gradient method finds the solution x in N = 2 steps, while the gradient descent method has to go through many more steps all orthogonal to each other before it finds the solution.

Find the A-orthogonal basis

The N A-orthogonal search directions {d_{0}, ⋯, d_{N−1}} can be constructed based on any set of N independent vectors {v_{0}, ⋯, v_{N−1}} by the Gram-Schmidt process:

d_{n} = v_{n} − ∑_{j=0}^{n−1} p_{d_{j}}(v_{n}) = v_{n} − ∑_{m=0}^{n−1} (\frac{d_{m}^{T}Av_{n}}{d_{m}^{T}Ad_{m}} ) d_{m} = v_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m} (2.125)

where β_{nm} = d_{m}^{T}Av_{n}/(d_{m}^{T}Ad_{m}) and β_{nm}d_{m} is the A-projection of v_{n} onto each of the previous direction d_{m}.

We will gain some significant computational advantage if we choose to use v_{n} = −g_{n}. Now the Gram-Schmidt process above becomes:

d_{n} = −g_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m},  (2.126)

where

β_{nm} = \frac{d_{m}^{T}Av_{n}}{d_{m}^{T}Ad_{m}} = −\frac{d_{m}^{T}Ag_{n}}{d_{m}^{T}Ad_{m}} (m < n)  (2.127)

The equation above indicates that g_{n} can be written as a linear combination of all previous search directions d_{0}, ⋯, d_{n}:

g_{n} = ∑_{i=0}^{n} α_{i}d_{i}  (2.128)

Pre-multiplying g_{n+1}^{T} on both sides, we get

g_{n+1}^{T}g_{k} = g_{n+1}^{T} (∑_{i=0}^{k} α_{i}d_{i}) = ∑_{i=0}^{k} α_{i} g_{n+1}^{T}d_{i} = 0  (2.129)

The last equality is due to g_{n+1}^{T}d_{k} = 0 in Eq. (2.124). We see that g_{n+1} is also orthogonal to all previous gradients g_{0}, ⋯, g_{n}.

Pre-multiplying g_{k}^{T} (k ≥ n) on both sides of Eq. (2.126), we get

g_{k}^{T} d_{n} = −g_{k}^{T} g_{n} − ∑_{m=0}^{n−1} β_{mn} g_{k}^{T} d_{m} = −g_{k}^{T} g_{n} = \{ \begin{matrix}−||g_{n}||^{2} & n = k \\ 0 & n < k \end{matrix} (2.130)

Note that all terms in the summation are zero as g_{k}^{T} d_{m} = 0 for all m < n ≤ k (Eq. (2.124)). Substituting g_{n}^{T} d_{n} = −||g_{n}||^{2} into Eq. (2.117), we get

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} Ad_{n}} = \frac{||g_{n}||^{2}}{d_{n}^{T} Ad_{n}}  (2.131)

Next we consider

g_{m+1} = Ax_{m+1}−b = A(x_{m}+δ_{m}d_{m})−b = (Ax_{m}−b)+δ_{m}Ad_{m} = g_{m}+δ_{m}Ad_{m}

(2.132)

Pre-multiplying g_{n}^{T} with n > m on both sides we get

g_{n}^{T} g_{m+1} = g_{n}^{T} g_{m} + δ_{m}g_{n}^{T} Ad_{m} = δ_{m}g_{n}^{T} Ad_{m}  (2.133)

where g_{n}^{T} g_{m} = 0 (m ≠ n). Solving for g_{n}^{T} Ad_{m} we get

g_{n}^{T} Ad_{m} = \frac{1}{δ_{m}} g_{n}^{T} g_{m+1} = \{ \begin{matrix}||g_{n}||^{2}/δ_{n−1} & m = n − 1 \\ 0 & m < n − 1 \end{matrix}  (2.134)

Substituting this into Eq. (2.127) we get

β_{nm} = −\frac{d_{m}^{T}Ag_{n}}{d_{m}^{T}Ad_{m}} = \{ \begin{matrix}−||g_{n}||^{2}/δ_{n−1}d_{n−1}^{T}Ad_{n−1} & m = n − 1 \\ 0 & m < n − 1 \end{matrix}  (2.135)

which is non-zero only when m = n − 1, i.e., there is only one non-zero term in the summation of the Gram-Schmidt formula for d_{n}. This is the reason why we choose v_{n} = −g_{n}. We can now drop the second subscript m in β_{nm}, and Eq. (2.125) becomes:

d_{n} = v_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m} = −g_{n} − β_{n}d_{n−1}  (2.136)

Substituting the step size δ_{n−1} = ||g_{n−1}||^{2}/d_{n−1}^{T}Ad_{n−1} (Eq. (2.131)) into the above expression for β_{nm} = β_{m}, we get

β_{n} = −\frac{||g_{n}||^{2}}{||g_{n−1}||^{2}}  (2.137)

We note that matrix A no longer appears in the expression.

The CG algorithm

Summarizing the above, we finally get the conjugate gradient algorithm in the following steps:

1. Set n = 0 and initialize the search direction (same as gradient descent):

d_{0} = −g_{0}  (2.138)

2. Terminate if the error ε = ||g_{n}||^{2} is smaller than a preset threshold. Otherwise, continue with the following:

3. Find optimal step size (Eq. (2.131)) and step forward

δ_{n} = \frac{||g_{n}||^{2}}{d_{n}^{T} Ad_{n}} ,  x_{n+1} = x_{n} + δ_{n}d_{n}  (2.139)

4. Update gradient:

g_{n+1} = \frac{d}{dx}f(x_{n+1})  (2.140)

5. Find coefficient for the Gram-Schmidt process (Eq. (2.137):

β_{n+1} = −\frac{||g_{n+1}||^{2}}{||g_{n}||^{2}}  (2.141)

6. Update search direction (Eq. (2.136)):

d_{n+1} = −g_{n+1} − β_{n+1}d_{n}  (2.142)

Set n = n + 1 and go back to step 2.

The algorithm above assumes the objective function f(x) to be quadratic with known A. But when f(x) is not quadratic, A is no longer available, we can still approximate f(x) as a quadratic function by the first three terms in its Taylor series in the neighborhood of its minimum. Also, the we need to modify the algorithm so that it does not depend on A. Specifically, the optimal step size δ_{n} calculated in step 3 above based on A can also be alternatively found by line minimization based on any suitable algorithms for 1-D optimization.

The Matlab code for the conjugate gradient algorithm is listed below:

function xn=myCG(j,tol)       % j is the objective function to minimize
    syms d;                   % variable for 1-D symbolic function f(d)
    x=symvar(j).';            % symbolic variables in objective function
    J=matlabFunction(j);      % the objective function
    G=jacobian(j).';          % symbolic gradient of objective function
    G=matlabFunction(G);      % the gradient function
    xn=zeros(length(x),1);    % initial guess of x
    xc=num2cell(xn);
    e=J(xc{:});               % initial error
    gn=G(xc{:});              % gradient at xn
    dn=-gn;                   % use negative gradient as search direction
    n=0;

while e>tol
    n=n+1;
    f=subs(j,x,xn+d*dn);          % convert n-D f(x) to 1-D f(d)
    delta=Opt1d(f);               % find delta that minimizes 1D function f(d)
    xn=xn+delta*dn;               % updata variable x
    xc=num2cell(xn);
    e=J(xc{:});                   % new error
    gn1=G(xc{:});                 % new gradient
    bt=-(gn1.'*gn1)/(gn.'*gn);    % find beta
    dn=-gn1-bt*dn;                % new search direction
    gn=gn1;                       % update gradient
    fprintf('%d\t(%.4f, %.4f, %.4f)\t%e\n',n,xn(1),xn(2),xn(3),e)
end
end

Here is the function that uses Newton’s method to find the optimal step size δ that minimizes the objective function as a 1-D function of δ:

function x=Opt1d(f)         % f is 1-D symbolic function to minimize
syms x;
tol=10^(-3);
d1=diff(f);             % 1st order derivative
d2=diff(d1);            % 2nd order derivative
f=matlabFunction(f);
d1=matlabFunction(d1);
d2=matlabFunction(d2);
x=0.0;                  % initial guess of delta
if d2(x)<=0             % second order derivative needs to be greater than 0
    x=rand-0.5;
end
y=x+1;
while abs(x-y) > tol    % minimization
    y=x;
    x=y-d1(y)/d2(y);    % Newton iteration
end
end

Example: To compare the conjugate method and the gradient descent method, consider

a very simple 2-D quadratic function

f(x, y) = x^{T} Ax = [x_{1}, x_{2}] [ \begin{matrix}3 & 1 \\ 1 & 2 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]

The performance of the gradient descent method depends significantly on the initial guess. For the specific initial guess of x_{0} = [1.5, −0.75]^{T}, the iteration gets into a zigzag pattern and the convergence is very slow, as shown below:

\begin{matrix}n & x = [x_{1}, x_{2}] & f(x) \\ 0 & 1.500000, −0.750000 & 2.812500 \\ 1 & 0.250000, −0.750000 & 0.468750e − 01 \\ 2 & 0.250000, −0.125000 & 7.812500e − 02 \\ 3 & 0.041667, −0.125000 & 1.302083e − 02 \\ 4 & 0.041667, −0.020833 & 2.170139e − 03 \\ 5 & 0.006944, −0.020833 & 3.616898e − 04 \\ 6 & 0.006944, −0.003472 & 6.028164e − 05 \\ 7 & 0.001157, −0.003472 & 1.004694e − 05 \\ 8 & 0.001157, −0.000579 & 1.674490e − 06 \\ 9 & 0.000193, −0.000579 & 2.790816e − 07 \\ 10 & 0.000193, −0.000096 & 4.651361e − 08 \\ 11 & 0.000032, −0.000096 & 7.752268e − 09 \\ 12 & 0.000032, −0.000016 & 1.292045e − 09 \\ 13 & 0.000005, −0.000016 & 2.153408e − 10 \end{matrix}

However, as expected, the conjugate gradient method takes exactly N = 2 steps from any initial guess to reach at the solution:

\begin{matrix}n & x = [x_{1}, x_{2}] & f(x) \\ 0 & 1.500000, −0.750000 & 2.812500e + 00 \\ 1 & 0.250000, −0.750000 & 4.687500e − 01 \\ 2 & 0.000000, −0.000000 & 1.155558e − 33 \end{matrix}

For an N = 3 example of f(x) = x^{T} Ax with

A = [ \begin{matrix}5 & 3 & 1 \\ 3 & 4 & 2 \\ 1 & 2 & 3 \end{matrix} ]

from an initial guess x_{0} = [1, 2, 3]^{T}, it takes the gradient descent method 41 iterations to reach x_{41} = [3.5486e − 06, −7.4471e − 06, 4.6180e − 06]^{T} corresponding to f(x) = 8.5429e − 11. From the same initial guess, it takes the conjugate gradient method only N = 3 iterations to converge to the solution:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & f(x) \\ 0 & 1.000000, 2.000000, 3.000000 & 4.500000e + 01 \\ 1 & −0.734716, −0.106441, 1.265284 & 2.809225e + 00 \\ 2 & 0.123437, −0.209498, 0.136074 & 3.584736e − 02 \\ 3 & −0.000000, 0.000000, 0.000000 & 3.949119e − 31 \end{matrix}

For an N = 9 example, it takes over 4000 iterations for the gradient descent method to converge with ||e|| ≈ 10^{−10}, but exactly 9 iterations for the CG method to converge with ||e|| ≈ 10^{−16}.

Example:

The figure below shows the search path of the conjugate gradient method applied to the minimization of the Rosenbrock function:

Example:

Solve the following N = 3 non-linear equation system by the CG method:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

The solution is known to be x_{1} = 0.5, x_{2} = 0, x_{3} = −0.5.

This equation system can be represented in vector form as f(x) = 0 and the objective function is J(x) = f^{T} (x)f(x). The iteration of the CG method with an initial guess x_{0} = 0 is shown below:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & J(x) \\ 0 & 0.0000, 0.0000, 0.0000 & 1.032500e + 02 \\ 1 & 0.0113, 0.0050, −0.5001 & 3.160163e + 00 \\ 2 & 0.0188, −0.0021, −0.5004 & 3.095894e + 00 \\ 3 & 0.5009, −0.0018, −0.5004 & 7.268252e − 05 \\ 4 & 0.5009, −0.0017, −0.5000 & 1.051537e − 05 \\ 5 & 0.5008, −0.0012, −0.5000 & 6.511151e − 06 \\ 6 & 0.5001, −0.0005, −0.5000 & 6.365321e − 07 \\ 7 & 0.5001, −0.0005, −0.5000 & 5.667357e − 07 \\ 8 & 0.5002, −0.0004, −0.5000 & 2.675128e − 07 \\ 9 & 0.5001, −0.0003, −0.5000 & 1.344218e − 07 \\ 10 & 0.5001, −0.0002, −0.5000 & 1.241196e − 07 \\ 11 & 0.5000, −0.0001, −0.5000 & 2.120969e − 08 \\ 12 & 0.5000, −0.0001, −0.5000 & 1.541814e − 08 \\ 13 & 0.5000, −0.0001, −0.5000 & 7.282025e − 09 \\ 14 & 0.5000, −0.0001, −0.5000 & 4.801781e − 09 \\ 15 & 0.5000, −0.0000, −0.5000 & 4.463926e − 09 \end{matrix}

In comparison, the gradient descent method would need to take over 200 iterations (with much reduced complexity though) to reach this level of error.

Conjugate gradient method used for solving linear equation systems: As discussed before, if x is the solution that minimizes the quadratic function

f(x) = x^{T} Ax/2 − b^{T} x + c, with A = A^{T} being symmetric and positive definite, it also satisfies d f(x)/dx = Ax − b = 0. In other words, the optimization problem is equivalent to the problem of solving the linear system Ax − b = 0, both can be solved by the conjugate gradient method.

Now consider solving the linear system Ax = b with A = A^{T}. Let d_{i}, (i = 1, ⋯, N) be a set of N A-orthogonal vectors satisfying d_{i}^{T} Ad_{j} = 0 (i ≠ j), which can be generated based on any N independent vectors, such as the standard basis vectors, by the Gram-Smidth method. The solution x of the equation Ax = b can be represented by these N vectors as

x = ∑_{i=1}^{N} c_{i}d_{i}  (2.143)

Now we have

b = Ax = A [∑_{i=1}^{N} c_{i}d_{i}] = ∑_{i=1}^{N} c_{i}Ad_{i}  (2.144)

Pre-multiplying d_{j}^{T} on both sides we get

d_{j}^{T} b = ∑_{i=1}^{N} c_{i}d_{j}^{T} Ad_{i} = c_{j}d_{j}^{T} Ad_{j}  (2.145)

Solving for c_{j} we get

c_{j} = \frac{d_{j}^{T} b}{d_{j}^{T} Ad_{j}}  (2.146)

Substituting this back into the expression for x we get the solution of the equation:

x = ∑_{i=1}^{N} c_{i}d_{i} = ∑_{i=1}^{N} (\frac{d_{i}^{T} b}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.147)

Also note that as b = Ax, the ith term of the summation above is simply the A-projection of x onto the ith direction d_{i}:

p_{d_{i}}(x) = (\frac{d_{i}^{T} Ax}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.148)

One application of the conjugate gradient method is to solve the normal equation to find the least-square solution of an over-constrained equation system Ax = b, where the coefficient matrix A is M by N with rank R = N < M. As discussed previously, the normal equation of this system is

A^{T} Ax = A^{T} b  (2.149)

Here A^{T} A is an N by N symmetric, positive definite matrix. This normal equation can be solved by the conjugate gradient method.

2.1.7 Issues of Local/Global Minimum

A main concern in optimization problems is whether the solution obtained by the algorithm is a local minimum or a global one. The simulated annealing (SA) algorithm is typically used to search for the global minimum in a high-dimensional space, by allowing up-hill moves in the iteration to avoid getting stuck by a local minimum. SA mimics the annealing process in metallurgy, in which the temperature is carefully controlled to decrease slowly so that the material reaches a state with minimum free energy.

The way the SA method avoids getting stuck at a local minimum is to allow the iteration steps to go up-hill. Specifically, the objective function f(x) evaluated at x_{n} is treated as the energy at x_{n}, and an up-hill move corresponding to an energy increase ΔE = f(x_{n+1}) − f(x_{n})_{0} can be accepted if some probability function is greater than a random threshold value T h within a certain range comparable with P:

P (ΔE, T) = e^{−ΔE/T} > T h  (2.150)

Obviously smaller ΔE and higher T will cause higher P, it is more probable for a move to be accepted.

During the entire search process, the temperature T gradually reduces so that the probability of accepting an up-hill move also gradually reduces. In the early stage of the search, it is more probable to go up-hill, thereby getting out of a local minimum; but toward the later part of the search process, such a move is less likely to happen, as presumably in this late stage of the search, the energy level is already low, not likely to be at a local minimum with high energy level. Through out the search, the best result corresponding to he lowest energy level is always updated, to be returned as the final solution at the end of the process.

2.2 Constrained Optimization

An optimization problem is more complicated if it is constrained, i.e., the arguments x of the objective function f(x) are subject to (abbreviated as s.t. in the following) certain equality or inequality constraints in terms of what values they can take. Such a generic constrained optimization problem can be formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & \{ \begin{matrix}h_{i}(x) = 0, (i = 1, ⋯ m) \\ g_{j}(x) ≤ 0, (j = 1, ⋯ n) \end{matrix} \end{matrix}  (2.151)

The set composed of all values of x that satisfy the m + n constraints is called the feasible region of the problem. The goal is to find a point x^{∗} in the feasible region at which f(x^{∗}) is an extremum.

In the most general case where the objective function f(x) and the constraint functions g_{i}(x) and h_{j}(x) are nonlinear, the process of solving this nonlinear optimization problem is called nonlinear programming (NLP). Specially, if the objective function is quadratic while the constraints are linear, (the feasible region is a polytope), the process is called quadratic programming (QP). More specially, if the objective function is linear as well as the constraints, the process is called linear programming (LP). In the following, we will first consider NLP in general, and then discuss more specifically QP and LP.

2.2.1 Optimization with Equality Constraints

The optimization problems subject to equality constraints can be generally formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & h_{i}(x) = h_{i}(x_{1}, ⋯, x_{N}) = 0, (i = 1, ⋯ m) \end{matrix}  (2.152)

where f(x) is the objective function and h_{i}(x) = 0 is the ith constraint. Here m ≤ N, as in general there does not exist a solution that satisfies more than N equations in the N-D space.

This problem can be visualized in the special case with N = 2 and m = 1, where both f(x) and h(x) are surfaces defined over the 2-D space spanned by x_{1} and x_{2}, and h(x) = 0 is the intersection line of h(x) and the 2-D plane. The solution x^{∗} must satisfy the following two conditions:

• x^{∗} is on the intersection line, i.e., h(x^{∗}) = 0;

• x^{∗} is on the contour line f(x) = d (for some d as shown in the figure), so that f(x^{∗}) neither increase nor decrease (no point is higher or lower) in the neighborhood of x^{∗} along the contour line.

For both conditions to be satisfied, the two contours h(x) = 0 and f(x) = d must coincide at x^{∗}, i.e., they must have the same tangents and therefore parallel gradients (perpendicular to the tangents):

▽_{x}f(x^{∗}) = λ ▽_{x} h(x^{∗})  (2.153)

where ▽_{x}f(x) = [∂f/∂x_{1}, ⋯, ∂f/∂x_{N}]^{T} is the gradient of f(x), and λ is a constant scaling factor which can be either positive if the two gradients are in the same direction or negative if they are in opposite directions. As here we are only interested in whether the two curves coincide or not, the sign of λ^{∗} can be either positive or negative.

To find such a point x^{∗}, we first construct a new objective function, called the Lagrange function (or simply Lagrangian):

L(x, λ) = f(x) − λ h(x)  (2.154)

where λ is the Lagrange multiplier, which can be either positive or negative (i.e., the sign in front of λ can be either plus or minus), and then set the gradient of this objective function with respect to λ as well as x to zero:

▽_{x,λ}L(x, λ) = ▽_{x,λ}[f(x) − λ h(x)] = 0  (2.155)

This equation contains two parts:

\{ \begin{matrix}▽_{x}f(x) = λ ▽_{x} h(x) \\ ▽_{λ}L(x, λ) = ∂L(x, λ)/∂λ = −h(x) = 0 \end{matrix}  (2.156)

This is an equation system containing N + m = 2 + 1 = 3 equations for the same number of unknowns x_{1}, x_{2}, λ:

\frac{∂f(x)}{∂x_{i}} = λ\frac{∂h(x)}{∂x_{i}} (i = 1, 2), h(x) = 0  (2.157)

The solution x^{∗} = [x_{1}^{∗}, x_{2}^{∗}]^{T} and λ^{∗} of this equation system happens to satisfy the two desired relationships: (a) ▽_{x}f(x) = λ ▽_{x} h(x), and (b) h(x^{∗}) = 0.

Specially, consider the following two possible cases:

• If λ^{∗} = 0, then ▽_{x}f(x^{∗}) = λ^{∗} ▽_{x} h(x^{∗}) = 0, indicating f(x^{∗}) is an extremum independent of the constraint, i.e., the constraint h(x) = 0 is inactive, and the optimization is unconstrained.

• If λ^{∗} ≠ 0, then ▽_{x}f(x^{∗}) ≠ 0 (as in general ▽_{x}h(x^{∗}) ≠ 0), indicating f(x^{∗}) is not an extremum without the constraint, i.e., the constraint is active, and the optimization is indeed constrained.

The discussion above can be generalized from 2-D to an N > 2 dimensional space, in which the optimal solution x^{∗} is to be found to extremize the objective f(x) subject to m equality constraints h_{i}(x) = 0 (i = 1, ⋯, m), each representing a contour surface in the N-D space. The solution x^{∗} must satisfy the following two conditions:

• x^{∗} is on all m contour surfaces so that h_{1}(x^{∗}) = ⋯ = h_{m}(x^{∗}) = 0, i.e., it is on the intersection of these surfaces, a curve in the space (e.g., the intersection of two surfaces in 3-D space is a curve);

• x^{∗} is on a contour surface f(x^{∗}) = d (for some value d) so that f(x^{∗}) is an extremum, i.e., it does not increase or degrease in the neighborhood of x^{∗} along the curve.

Combining these conditions, we see that the intersection (a curve or a surface) of the m surfaces h_{i}(x) = 0 must coincide with the contour surface f(x) = d at x^{∗}, i.e., the intersection of the tangent surfaces of h_{i}(x) = 0 through x^{∗} must coincide with the tangent surface of f(x) = d at x^{∗}, therefore the gradient ▽f(x^{∗}) must be a linear combination of the m gradients ▽h_{i}(x^{∗}):

▽_{x}f(x^{∗}) = ∑_{i=1}^{m} λ_{i}^{∗} ▽_{x} h_{i}(x^{∗})  (2.158)

To find such a solution x^{∗} of the optimization problem, we first construct the Lagrange function:

L(x, λ) = f(x) − ∑_{i=1}^{m} λ_{i}h_{i}(x)  (2.159)

where λ = [λ_{1}, ⋯, λ_{m}]^{T} is a vector for m Lagrange multipliers, and then set its gradient to zero:

▽_{x,λ}L(x, λ) = ▽_{x,λ} [f(x) − ∑_{i=1}^{m} λ_{i}h_{i}(x)] = 0  (2.160)

to get the following two equation systems of N and m equations repectively:

▽_{x}f(x) = ∑_{i=1}^{m} λ_{i} ▽_{x} h_{i}(x) i.e. \frac{∂f(x)}{∂x_{j}} = ∑_{i=1}^{m} λ_{i} \frac{∂h_{i}(x)}{∂x_{j}} (j = 1, ⋯, N),

(2.161)

and

\frac{∂L(x, λ)}{∂λ_{i}} = h_{i}(x) = 0 (i = 1, ⋯, m)  (2.162)

The first set of N equations indicates that the gradient ▽_{x}f(x^{∗}) at x^{∗} is a linear combination of the m gradients ▽_{x}h_{i}(x^{∗}), while the second set of m equations guarantees that x^{∗} also satisfy the m equality constraints. Solving these equations we get the desired soluton x^{∗} together with the Lagrange multipliers λ_{1}, ⋯, λ_{m}. This is the Lagrange multiplier method.

Specially, if λ_{i} = 0, then the ith constraint is not active. More specially if λ_{i} = 0 for all i = 1, ⋯, m, then

▽_{x}f(x^{∗}) = ∑_{i=1}^{m} λ_{i} ▽_{x} h_{i}(x) = 0  (2.163)

indicating f(x^{∗}) is an extremum independent of any of the constraints, i.e., x^{∗} is an unconstrained solution, and the optimization problem is unconstrained.

2.2.2 Optimization with Inequality Constraints

The optimization problems subject to inequality constraints can be generally formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & g_{j}(x) = g_{j}(x_{1}, ⋯, x_{N}) ≤ or ≥ 0,  (j = 1, ⋯ n) \end{matrix} (2.164)

Again, to visualize the problem we first consider an example with N = 2 and n = 1, as shown in the figure below for the minimization (left) and maximization (right) of f(x) subject to g(x) > 0. The constrained solution x^{∗} = [x_{1}^{∗}, x_{2}^{∗}]^{T} is on the boundary of the feasible region satisfying g(x_{1}, x_{2}) = 0, while the unconstrained extremum is outside the feasible region.

Consider the following two possible cases.

• First, if the unconstrained extremum at which ▽_{x}f(x) = 0 is outside the feasible region, i.e., the inequality constraint is active, then the constrained solution x^{∗} must be

– on the boundary of the feasible region, i.e., g(x^{∗}) = 0,

– different from the unconstrained solution, i.e., ▽_{x}f(x^{∗}) ≠ 0; The problem becomes the same as an equality constrained problem considered before, which requires the objective function f(x) and the constraining function g(x) to have the same tangent at x^{∗}, or parallel gradients:

▽_{x}f(x^{∗}) = μ^{∗} ▽_{x} g(x^{∗}) or ▽_{x} f(x^{∗}) − μ^{∗} ▽_{x} g(x^{∗}) = 0 (2.165)

However, different from before, now we are also concerned with whether the two gradients have the same or opposite directions, corresponding to μ > 0 or μ < 0, respectively, as illustrated in the 1-D examples in the figure below:

Depending on whether f(x) is to be maximized or minimized, and whether the constraint is g(x) ≥ 0 or g(x) ≤ 0, there exist four possible cases in terms of the sign of μ, as summarized in the table below:

g(x) ≥ 0  g(x) ≤ 0
max f(x) (1) ▽ f(x_{1}) = μ ▽ g(x_{1}), μ < 0 (3) ▽ f(x_{3}) = μ ▽ g(x_{3}), μ > 0
min f(x) (2) ▽ f(x_{2}) = μ ▽ g(x_{2}), μ > 0 (4) ▽ f(x_{4}) = μ ▽ g(x_{4}), μ < 0

(2.166)

Case max/min subject to  μ
(1) max f(x) g(x) ≥ 0 μ < 0
(2) min f(x) g(x) ≥ 0 μ > 0
(3) max f(x) g(x) ≤ 0 μ > 0
(4) min f(x) g(x) ≤ 0 μ < 0

(2.167)

We can also consider the product of μ and g(x) for either maximization or minimization:

μ g(x) \{ \begin{matrix}≤ 0 & for maximization \\ ≥ 0 & for minimization \end{matrix}  (2.168)

• Second, if the unconstrained extremum is inside the feasible region, i.e., the inequality constraint is inactive, then the problem is actually unconstrained and the results above are no longer valid. The solution x^{∗} of this unconstrained problem is

– the same as the unconstrained solution, i.e., ▽_{x}f(x^{∗}) = 0;

– not on the boundary of the feasible region, i.e., g(x^{∗}) ≠ 0; Now we need to assume μ = 0 and find the solution solving the same equation ▽_{x}f(x) = μ ▽_{x} g(x) = 0

Summarizing the two cases above, we see that g(x^{∗}) = 0 but μ^{∗} ≠ 0 in the first case, g(x^{∗}) ≠ 0 but μ^{∗} = 0 in the second case, i.e., the following holds in either case:

μ^{∗} g(x^{∗}) = 0  (2.169)

The discussion above can be generalized from 2-D to N > 2 dimensional space, in which the optimal solution x^{∗} is to be found to extremize the objective f(x) subject to n inequality constraints g_{i}(x) = 0 (i = 1, ⋯, n). To solve this inequality constrained optimization problem, we first construct the Lagrangian:

L(x, μ) = f(x) − ∑_{i=1}^{n} μ_{i} g_{i}(x)  (2.170)

We note that in some literatures, a plus sign is used in front of the summation of the second term. This is equivalent to our discussion here so long as the sign of μ indicated in Table 2.167 is negated.

We now set the gradient of the Lagrangian to zero:

▽_{x,μ}L(x, μ) = ▽_{x,μ} [f(x) − ∑_{i=1}^{n} μ_{i} g_{i}(x)] = 0  (2.171)

and get two equation systems of N and n equations, respectively:

▽_{x}f(x) = ∑_{i=1}^{n} μ_{i} ▽_{x} g_{i}(x)  (2.172)

and

\frac{∂L(x, μ)}{∂μ_{i}} = g_{i}(x) = 0 (i = 1, ⋯, n)  (2.173)

The result above for the inequality constrained problems is the same as that for the equality constrained problems considered before. However, we note that there is an additional requirement regarding the sign of the scaling coifficients. For an equality constrained problem, the direction of the gradient ▽g(x) is of no concern, i.e., the sign of λ is unrestricted; but here for an inequality constrained problem, the sign of μ needs to be consistent with those shown in Table 2.167, otherwise the constraints may be inactive.

As a simple example, consider the iterative minimization of a function f(x). To find the best increment Δx that minimizes f(x + Δx) ≈ f(x) + g_{f}(x)^{T} Δx but at the same time is also subjected to the constraint of limited length, i.e., ||Δx||^{2}/2 ≤ C, we first construct the Lagrangian:

L(x, λ) = f(x) + g_{f}x^{T} Δx) − μ(||Deltax||^{2}/2 − C)  (2.174)

and set its derivative to zero:

\frac{d}{dΔx}L(x, λ)g_{f}(x) − μΔx = 0  (2.175)

where μ < 0 according to Table 2.167. We then solve the equation to get

Δx = \frac{1}{μ}g_{f}(x)  (2.176)

and the iteration

x_{n+1} = x_{n} − δg(x_{n})  (2.177)

same as the gradient descent method in Eq. (2.25).

We now consider the general optimization of an N-D objective function f(x) subject to multiple constraints of both equalities and inequalities:

max/min: f(x) = f(x_{1}, ⋯, x_{N})
s. t.:  \{ \begin{matrix}h_{i}(x) = 0, (i = 1, ⋯ m) \\ g_{j}(x) ≤ 0 or g_{j}(x) ≥ 0,  (j = 1, ⋯ n) \end{matrix} (2.178)

For notational convenience, we represent the m + n equality and inequality constraints in vector form as:

max/min: f(x)
s. t.:  \{ \begin{matrix}h(x) = 0 \\ g(x) ≤ 0 or g(x) ≥ 0 \end{matrix}

(2.179)

where h(x) = [h_{1}(x), ⋯, h_{m}(x)]^{T} and g(x) = [g_{1}(x), ⋯, g_{n}(x)]^{T}.

To solve this optimization problem, we first construct the Lagrangian

L(x, λ, μ) = f(x)−∑_{i=1}^{m} λ_{i}h_{i}(x)−∑_{j=1}^{n} μ_{j}g_{j}(x) = f(x)−λ^{T} h(x)−μ^{T} g(x) (2.180)

where the Lagrange multipliers in λ = [λ_{1}, ⋯, λ_{m}]^{T} and μ = [μ_{1}, ⋯, μ_{n}]^{T} are for the m equality and n non-negative constraints, respectively. We then set to zero the gradient of L(x, λ, μ) with respect to both λ and μ as well as x. The solution x^{∗} can then be obtained by solving the resulting equation system. While λ_{i} can be either positive or negative, with sign of μ_{j} needs to be consistent with those specified in Table 2.167. Otherwise the inequality constraints is inactive.

Example:

Find the extremum of f(x_{1}, x_{2}) = x_{1}^{2} +x_{2}^{2} subject to each of the three different constraints: x_{1} + x_{2} = 1, x_{1} + x_{2} ≤ 1, and x_{1} + x_{2} ≥ 1.

•

\{ \begin{matrix}min/max: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & h(x_{1}, x_{2}) = x_{1} + x_{2} − 1 = 0 \end{matrix}

The Lagrangian is:

L(x_{1}, x_{2}, λ) = f(x_{1}, x_{2}) − λg(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} − λ(x_{1} + x_{2} − 1)

Solving the following equations

\frac{∂L}{∂x_{1}} = 2x_{1} − λ = 0, \frac{∂L}{∂x_{2}} = 2x_{2} − λ = 0, \frac{∂L}{∂λ} = x_{1} + x_{2} − 1 = 0

we get λ^{∗} = 1, x_{1}^{∗} = x_{2}^{∗} = 0.5, i.e., the function is minimized at f(0.5, 0.5) = 0.5. We note that f(x_{1}, x_{2}) and g(x_{1}, x_{2}) have the same gradients:

▽f(0.5, 0.5) = ▽h(0.5, 0.5) = [1, 1]^{T}

• Consider these two problems:

\{ \begin{matrix}min: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≥ 0 \end{matrix}  \{ \begin{matrix}max: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≤ 0 \end{matrix}

These problems are cases 2 and 3 in Table 2.167. They have the same Lagrangian as in the previous case and therefore the same results x_{1}^{∗} = x_{2}^{∗} = 0.5 and μ^{∗} = 1 > 0. According to Table 2.167, x^{∗} is the solution for the minimization problem subject to g(x) ≥ 0 (left), or the maximization problem subject to g(x) ≤ 0.

• Consider these two problems:

\{ \begin{matrix}max: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≥ 0 \end{matrix}  \{ \begin{matrix}min: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≤ 0 \end{matrix}

These problems are cases 1 and 4 in Table 2.167. They have the same Lagrangian as in the previous case and therefore the same results x_{1}^{∗} = x_{2}^{∗} = 0.5, and μ^{∗} = 1 > 0. According to Table 2.167, x^{∗} is not the solution of either the maximization problem subject to g(x) ≥ 0 (left), or the minimization problem subject to g(x) ≤ 0 (right), i.e., the constraint is inactive. We therefore need to assume μ^{∗} = 0 and solve the following equations for an unconstrained problem:

\frac{∂L}{∂x_{1}} = 2x_{1} = 0, \frac{∂L}{∂x_{2}} = 2x_{2} = 0

Now we get x_{1}^{∗} = x_{2}^{∗} = 0 and x^{∗} = 0, with minimum f(x^{∗}) = 0. This is the solution for the minimization problem (right), at which g(x^{∗}) ≠ 0, i.e., the constraint is inactive. However this is not the solution for the maximization problem (left) as the function is not bounded from above.

2.2.3 Duality and KKT Conditions

Previously we consider the constrained minimization problem:

\{ \begin{matrix}min: & f_{p}(x) \\ s. t.: & h(x) = 0,  g(x) ≤ or ≥ 0 \end{matrix}  (2.181)

with its Lagrangian:

L(x, λ, μ) = f_{p}(x) − λ^{T} h(x) − μ^{T} g(x)  (2.182)

The optimal solution x^{∗} can be found by solving the following equation

▽_{x}L(x, λ, μ) = 0  (2.183)

at which the objective function reaches its minimum p^{∗} = f_{p}(x^{∗}).

In particular, if the objective function f_{p}(x) and all inequality constraints g_{i}(x) are convex and the equality constraints h_{i}(x) are linear (affine), i.e., the feasible region is convex (Section A.24), then the problem is a convex optimization with some nice properties as shown below.

This problem is called the primal problem, as there also exists an equivalent dual problem, which may be easier to solve in certain sense. We first define the dual function as the infimum (lower bound) of the Lagrangian to be minimized:

f_{d}(λ, μ) = inf_{x} L(x, λ, μ) = inf_{x} [f_{p}(x) − λ^{T} h(x) − μ^{T} g(x)]  (2.184)

Furthermore, to find the tightest lower bound, the supremum (upper bound) of the dual function f_{d}(λ, μ) over its variables λ and μ, we solve the following dual problem:

\{ \begin{matrix}max: & f_{d}(λ, μ) = inf_{x} L(x, λ, μ) \\ s. t.: & μ ≤ or ≥ 0 \end{matrix}  (2.185)

Here λ for the equality constrains of the primal problem can be either positive or negative, i.e., it is not constrained in the dual problem. However, the sign of μ for the inequalities constraint of the primal minimization problem needs to satisfy the following according to Eq. (2.168) (the second inequality for this minimization primal problem):

μ^{T} g(x) ≥ 0  (2.186)

i.e., μ is subject to the inequality constraint of the dual problem. We denote the optimal solution of the dual problem by {λ^{∗}, μ^{∗}}, and the maximum of the objective function by d^{∗} = f_{d}(λ^{∗}, μ^{∗}).

Obviously the primal objective function of any feasible x is always no less than its lower bound, the dual objective function of any feasible {λ, μ}:

f_{p}(x) ≥ f_{d}(λ, μ)  (2.187)

and this inequality also holds for the optimal solutions x^{∗} and {λ^{∗}, μ^{∗}}:

p^{∗} = f_{p}(x) ≥ f_{d}(λ, μ) = d^{∗}  (2.188)

The difference p^{∗} −d^{∗} is defined as the duality gap. Obviously p^{∗} −d^{∗} ≥ 0 always holds and the duality is called weak duality. In some problems where certain conditions (Slater’s conditions) are satisfied, p^{∗} = d^{∗}, i.e., d^{∗} is the tightest lower bound, the reality is called strong duality.

We now consider the necessary and sufficient conditions for the solutions of both the primal and dual problems. We first consider the Karush-Kuhn-Tucker (KKT) conditions:

• Stationarity: x^{∗} maximizes or minimizes the Lagrangian, i.e.,

▽_{x}L(x^{∗}, λ, μ) = 0  (2.189)

• Primal feasibility: x^{∗} must be inside the feasible region:

h_{i}(x^{∗}) = 0, g_{j}(x^{∗}) ≤ or ≥ 0,  (i = 1, ⋯, m, j = 1, ⋯, n) (2.190)

• Dual feasibility: All multipliers in μ for the inequalities must take proper sign as specified by Table 2.167:

μ_{j}^{∗} ≤ or ≥ 0, (j = 1, ⋯, n)  (2.191)

• Complementarity: Either μ_{j}^{∗} or g_{j}(x^{∗}) must be zero depending on whether the jth constraint is inactive or not:

– if inactive, then g_{j}(x^{∗}) ≠ 0, but μ_{j}^{∗} = 0;

– if active, then μ_{j}^{∗} ≠ 0, but g_{j}(x^{∗}) = 0.

Combining both cases, we get

(μ^{∗})^{T} g(x^{∗}) = 0  (2.192)

We then intruduce the following theorem regarding the KKT conditions and the solutions of the primal and dual problems.

Theorem:

• Necessary conditions: If x^{∗} and {λ^{∗}, μ^{∗}} are solutions for the respective primal and dual problems with strong duality, then they satisfy all KKT conditions.

• Sufficient conditions: If x^{∗} and {λ^{∗}, μ^{∗}} satisfy all KKT conditions, then they are solutions for the respective primal and dual problems, and strong duality holds, when the primal objective function f_{p}(x^{∗}) and the inequality constraints g_{j}(x) are convex, and equality constraints h_{j}(x) are linear.

Proof:

• Necessary conditions: If x^{∗} and {λ^{∗}, μ^{∗}} are respectively the primal and dual solutions, then the primal and dual feasibilities must be satisfied, and so is stationarity as x^{∗} as the optimal solution satisfies Eq. (2.183). In addition, we also have h(x^{∗}) = 0. To show complementarity also holds, consider

d^{∗} = f_{d}(λ^{∗}, μ^{∗}) = inf_{x} L(x, λ^{∗}, μ^{∗})
= inf_{x} [f_{p}(x) − (λ^{∗})^{T} h(x) − (μ^{∗})^{T} g(x)]
≤ f_{p}(x^{∗}) − (λ^{∗})^{T} h(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
= f_{p}(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
≤ f_{p}(x^{∗}) = p^{∗}  (2.193)

The last inequality is due to Eq. (2.186). But as d^{∗} = p^{∗} due to the strong duality, the inequalities need to be equalities, i.e., (μ^{∗})^{T} g(x^{∗}) = 0 and complementarity holds.

• Sufficient conditions: If x^{∗} and {λ^{∗}, μ^{∗}} satisfy the KKT conditions, then they are feasible for the respective primal and dual problems. Also, as f_{p}(x) and all g_{j}(x) are convex and h_{j}x) are linear, Lagrangian in Eq. (2.182) as a sum of convex functions is also convex (see Section A.24), and stationarity in Eq. (2.189) indicates x^{∗} is a global minimum, therefore

d^{∗} = f_{d}(λ^{∗}, μ^{∗}) = L(x^{∗}, λ^{∗}, μ^{∗}) = f_{p}(x^{∗}) − (λ^{∗})^{T} h(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
= f_{p}(x^{∗}) = p^{∗}  (2.194)

where the second line is due to the KKT conditions in Eqs. (2.190) and (2.192).

In summary, we see that KKT conditions are necessary and satisfied by all solutions of constrained optimization problems, and they can also be sufficient for the solution of such problems if the primal problem further satisfies some additional conditions of convexity. In such a case, strong duality holds and the solution of the dual problem is the tightest lower bound of the primal, same as the solution of the primal.

Consider, as an example of the inequality constrained minimization problems, the algorithm of support vector machine (SVM) for binary classification (Section 13). To find the decision boundary that optimally saperates two classes, we need to solve the following minimization problem constrained by a set of inequalities imposed by the data samples {(x_{n}, y_{n}), n = 1, ⋯, N} in the training set:

\{ \begin{matrix}min: & \frac{1}{2}||w||^{2} \\ s. t.: & y_{n}(x_{n}^{T} w + b) ≥ 1 (n = 1, ⋯, N) \end{matrix}  (2.195)

where w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}. We note that the quadratic objective function is convex and so are the affine inequality constraints, i.e., this is a convex problem. The solution w with minimum norm ||w||^{2} subject to the inequality constraits can be found by solving the dual problem in terms of the Lagrange multipliers in α = [α_{1}, ⋯, α_{N}]^{T}, subject to certain inequality constraints (detailed derivation in Section 13):

\{ \begin{matrix}max: & ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m}) \\ s. t.: & ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N) \end{matrix}  (2.196)

As the dual function can be written a quadratic form:

α^{T} 1 − \frac{1}{2}α^{T} Qα  (2.197)

where Q is an N × N symmetric matrix of which the component in the mth row and nth column is Q(m, n) = y_{m}y_{n}x_{m}^{T}x_{n}, this is a quadratic programming problem, which can be solved by the interior point algorithm to be discussed in Section 2.2.7.

2.2.4 Linear Programming (LP)

The basic problem in linear programming (LP) is to minimize/maximize a linear objective function f(x_{1}, ⋯, x_{N}) = ∑_{i=1}^{N} c_{i}x_{i} of N variables x_{1}, ⋯, x_{N} under a set of M linear constraints:

\begin{matrix}max: & f(x_{1}, ⋯, x_{N}) = ∑_{i=1}^{N} c_{i}x_{i} \\ s. t.: & \{ \begin{matrix}∑_{i=1}^{N} a_{1i}x_{i} ≤ b_{1} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ ∑_{i=1}^{N} a_{Mi}x_{i} ≤ b_{M} \\ x_{1} ≥ 0, ⋯, x_{N} ≥ 0 \end{matrix} \end{matrix}  (2.198)

The LP problem can be more concisely represented in matrix form:

max: c^{T} x
s. t.: \{ \begin{matrix}Ax − b ≤ 0 \\ x ≥ 0 \end{matrix} (2.199)

where

x = [ x_{1} ; ⋮ ; x_{N} ] c = [ c_{1} ; ⋮ ; c_{N} ] b = [ b_{1} ; ⋮ ; b_{M} ] A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ]

(2.200)

and the smaller or greater than sign in a vector inequality should be understood as an element-wise relationship between the corresponding elements of the vectors on both sides.

For example, the objective function could be the total profit, the constraints could be some limited resources. Spicifically, x_{1}, ⋯, x_{N} may represent the quantities of N different types of products, c_{1}, ⋯, c_{N} represent their unit prices, and a_{ij} represent the consumption of the ith resource by the jth product, the goal is to maximize the profit f(x_{1}, ⋯, x_{N}) subject to the constraints imposed by the limited resources b_{1}, ⋯, b_{M}.

Here all variables are assumed to be non-negative. If there exists a variable that is not restricted, it can be eliminated. Specifically, we can solve one of the constraining equations for the variable and use the resulting expression of the variable to replace all its appearances in the problem. For example:

\begin{matrix}max: & f(x_{1}, x_{2}, x_{3}) = 2x_{1} − x_{2} + 3x_{3} \\ s. t.: & \{ \begin{matrix}x_{1} − 2x_{2} + x_{3} = 3 \\ 3x_{1} − x_{2} + 4x_{3} = 10 \\ x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix} \end{matrix}  (2.201)

Solving the first constraining equation for the free variable x_{1}, we get x_{1} = 3+2x_{2}−x_{3}. Substituting this into the objective function and the other constraint, we get 2x_{1} − x_{2} + x_{3} = 3x_{2} + x_{3} + 6 and 5x_{2} + x_{3} = 1, the problem can be reformulated as:

\begin{matrix}max: & f(x_{2}, x_{3}) = 3x_{2} + x_{3} + 6 \\ s. t.: & \{ \begin{matrix}5x_{2} + x_{3} = 1 \\ x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix} \end{matrix}  (2.202)

Given the primal LP problem above, we can further find its dual problem. We first construct the Lagrangian of primal problem:

L(x) = c^{T} x − y^{T} (Ax − b) = (c − A^{T} y)^{T} x + y^{T} b  (2.203)

where y = [y_{1}, ⋯, y_{M}]^{T} contains the Lagrange multipliers for the inequality constraints Ax − b ≤ 0, and y ≥ 0 according to Table 2.167. We define the dual function as the maximum of L(x) over x:

f_{d}(y) = max_{x} L(x) = max_{x}[c^{T} x − y^{T} (Ax − b)] = max_{x}[(c − A^{T} y)^{T} x + y^{T} b]

(2.204)

To find the value of x that maximizes L(x), we set its gradient to zero and get:

▽_{x}L(x) = ▽_{x}[(c − A^{T} y)^{T} x + y^{T} b] = c − A^{T} y = 0  (2.205)

Substituting this into L_{d}(y) = max_{x} L(x) we get: f_{d}(y) = y^{T} b, which is the upper bound of f_{p}(x), under the constraint that c − A^{T} y ≤ 0 so that (c − A^{T} y)^{T} x ≤ 0 contributes negatively to L(x) (as x ≥ 0):

f_{d}(y) = b^{T} y ≥ L(x) = c^{T} x − y^{T} (Ax − b) ≥ c^{T} x = f_{p}(x)  (2.206)

This upper bound needs to be minimized with respect to y, to become the tightest upper bound, i.e., the original primal problem of maximization in Eq. (2.199) has now been converted into the following dual problem of minimization:

\{ \begin{matrix}max : & c^{T} x \\ s.t. & Ax − b ≤ 0, x ≥ 0 \end{matrix} ⟺ \{ \begin{matrix}min & b^{T} y \\ s.t. & A^{T} y − c ≥ 0, y ≥ 0 \end{matrix} (2.207)

Following the same approach, we can also find the dual of a minimization primal problem:

\{ \begin{matrix}min : & c^{T} x \\ s.t. & Ax − b ≤ 0, x ≥ 0 \end{matrix} ⟺ \{ \begin{matrix}max & b^{T} y \\ s.t. & A^{T} y − c ≤ 0, y ≤ 0 \end{matrix}
⟺ \{ \begin{matrix}max & −b^{T} y \\ s.t. & A^{T} y + c ≥ 0, y ≥ 0 \end{matrix}(2.208)

Moreover, we can show that the dual of a dual problem is the original primal problem.

If either the primal or the dual is feasible and bounded, so is the other, and they form a strong duality, the solution d^{∗} of the dual problem is the same as the solution p^{∗} of the primal problem. Also, as the primal and dual problems are completely symmetric.

An inequality constrained LP problem can be converted to an equality constrained problem by introducing slack variables:

∑_{i=1}^{N} a_{i}x_{i} ≤ b ⟹ ∑_{i=1}^{N} a_{i}x_{i} + s = b, (s ≥ 0)  (2.209)

so that it can be reformulated to take the standard form:

max: z = f(x) = c^{T} x = ∑_{j=1}^{N} c_{j}x_{j}

s. t.: \{ \begin{matrix}∑_{i=1}^{N} a_{1i}x_{i} & +s_{1} & & & & = b_{1} \\ ∑_{i=1}^{N} a_{2i}x_{i} & & +s_{2} & & & = b_{2} \\ ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ \\ ∑_{i=1}^{N} a_{Mi}x_{i} & & & & +s_{M} & = b_{M} \\ x_{1} ≥ 0, & ⋯, & x_{N} ≥ 0, & s_{1} ≥ 0, & ⋯ & s_{M} ≥ 0 \end{matrix} (2.210)

We further redefine the following:

• x is an N + M dimensional augmented variable vector that includes the M slack variables {s_{1}, ⋯, s_{M}} as well as the N original variables {x_{1}, ⋯, x_{N}}:

x = [x_{1}, x_{2}, ⋯, x_{N}, s_{1}, s_{2}, ⋯, s_{M}]^{T}  (2.211)

• A is redefined as an M × (N + M) augmented coefficient matrix that includes coefficients for both types of variables:

A = [ \begin{matrix}a_{11} & a_{12} & ⋯ & a_{1N} & 1 & 0 & ⋯ & 0 \\ a_{21} & a_{22} & ⋯ & a_{1N} & 0 & 1 & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ & ⋮ & ⋮ & ... & ⋮ \\ a_{M1} & a_{M2} & ⋯ & a_{MN} & 0 & ⋯ & 0 & 1 \end{matrix} ]_{M×(N+M)} = [ A_{M×N} | I_{M×M} ]

(2.212)

where A_{M×N} are the coefficients for the N original variables {x_{1}, ⋯, x_{N}}, and the identity matrix I_{M×M} = [e_{1}, ⋯, e_{M}] is for the unity coefficients of the M slack variables {s_{1}, ⋯, s_{M}}, each of which appears in the M constraints only once.

Now the LP problem can be expressed in the standard form (original form on the left, with x and A redefined on the right):

max: c^{T} x s. t.: \{ \begin{matrix}Ax + s = b \\ x ≥ 0, s ≥ 0 \end{matrix}  or  max: c^{T} x s. t.: \{ \begin{matrix}Ax = b \\ x ≥ 0 \end{matrix}  (2.213)

An LP problem can be viewed geometrically. If normalize c so that ||c|| = 1, then the objective function f(x) = c^{T} x becomes the projection of x onto c. Each of the M equality constraints in Ax = b represents a hyper-plane in the N-D vector space perpendicular to its normal direction a_{j} = [a_{i1}, ⋯, a_{IN}]^{T}:

∑_{j=1}^{N} a_{ij}x_{j} = a_{j}^{T} x = b_{i},  (i = 1, ⋯, M)  (2.214)

Also, each of the N non-negativity conditions x_{j} ≥ 0 in x ≥ 0 corresponds to a hyper-plane perpendicular to the jth standard basis vector e_{j}. In general, in an N-D space, if none of the hyper-planes is parallel to any others, then no more than N hyper-planes intersect at one point. For example, in a 2-D or 3-D space, two straignt lines or three surfaces intersect at a point. Here for the LP problem in the N-D space, the total number of intersection points formed by the N + M hyper-planes is

C_{M+N}^{N} = C_{M+N}^{M} = \frac{(M + N)!}{N! M!}  (2.215)

The polytope enclosed by these N + M hyper-planes is called the feasible region, in which the optimal solution must lie. The vertices of the polytopic feasible region are a subset of the C_{M+N}^{N} intersections.

Fundamental theorem of linear programming

The optimal solution x^{∗} of a linear programming problem formulated above is either a vertex of the polytopic feasible region P, or lies on a hyper-surface of the polytope, on which all points are optimal solutions.

Proof:

Assume the optimal solution x^{∗} ∈ P is interior to the polytopic feasible region P, then there must exist some ϵ > 0 such that the hyper-sphere of radius ϵ centered at x^{∗} is inside P. Evaluate the objective function at a point on the hyper-sphere

x′ = x^{∗} + ϵ c/||c|| ∈ P  (2.216)

we get

f(x′) = c^{T} x′ = c^{T} (x^{∗} + ϵ c/||c||)
= c^{T} x^{∗} + ϵ c^{T} c/||c|| = c^{T} x^{∗} + ϵ ||c|| > c^{T} x^{∗} = f(x^{∗}) (2.217)

i.e., x^{∗} cannot be the optimal solution as assumed. This contradiction indicates that an optimal solution x^{∗} must be on the surface of P, either at one of its vertices or on one of its surfaces. Q.E.D.

Based on this theorem, the optimization of a linear programming problem could be solved by exhaustively checking each of the C_{N+M}^{N} intersections formed by N of the N + M hyper-planes to find (a) whether it is feasible (satisfying all N + M constraints), and, if so, (b) whether the objective function f(x) is maximized at the point. Specifically, we choose N of the N +M equations for the the constraints and solve this N-equation and N-unknown linear system to get the intersection point of N corresponding hyper-planes. This brute-force method is most straight forward, but the computational complexity is high when N and M, and therefore C_{N+M}^{N}, become large.

Example:

max: f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2}_{s. t.: \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix}}

or

max: f_{p}(x) = c^{T} x
s. t.: \{ \begin{matrix}Ax − b ≤ 0 \\ x ≥ 0 \end{matrix}

where

x = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , c = [ \begin{matrix}2 \\ 3 \end{matrix} ] , b = [ \begin{matrix}18 \\ 60 \\ 40 \end{matrix} ] , A = [ \begin{matrix}2 & 1 \\ 6 & 5 \\ 2 & 5 \end{matrix} ]

This problem has N = 2 variables with the same number of non-negativity constraints and M = 3 linear inequality constraints. The n + m = 5 straight lines form C_{N+M}^{N} = C_{5}^{2} = 10 intersections, out of which 5 are the vertices of the polygonal feasible region satisfying all constraints. The value of the objective function f_{p}(x_{1}, x_{2}) = c^{T} x = c_{1}x_{1} + c_{2}x_{2} is proportional to the projection of the 2-D variable vector x = [x_{1}, x_{2}]^{T} onto the coefficient vector c = [2, 3]^{T}. The goal of this linear programming problem is to find a point x inside the polygonal feasible region with maximum f(x) = c^{T} x (the projection of x onto vector c if ||c|| = 1). Geometrically, this can be understood as to push the hyper-plane c^{T} x = z, here a line in 2-D space, its normal direction c, away from the origin along to eventually arrive at the vertex of the polytopic feasible region farthest away from the origin.

This LP problem can be converted into the standard form:

\begin{matrix}max: & f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix} \end{matrix}  ⟹  \begin{matrix}max: & f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} + s_{1} = 18 \\ 6x_{1} + 5x_{2} + s_{2} = 60 \\ 2x_{1} + 5x_{2} + s_{3} = 40 \\ x_{1} ≥ 0, x_{2} ≥ 0, s_{1} ≥ 0, s_{2} ≥ 0, s_{3} ≥ 0 \end{matrix} \end{matrix}

Listed in the table below are the C_{M+N}^{N} = C_{5}^{2} = 10 intersections x, called basic solutions, together with the corresponding slack variables s_{1}, s_{2}, s_{3} and the objective function value f(x) (proportional to the projection of x onto c). We see that out of the ten basic solutions, five are feasible (vertices of the polytopic feasible region) with all M + N = 5 variables taking non-negative values, while the remaining five are not feasible, as some of the slack variables are negative, and thereby violating the constraints. Out of the five feasible solutions, the one at x^{∗} = (5, 6) is optimal with maximum objective function value f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} = 2 × 5 + 3 × 6 = 28.

x_{1} x_{2} s_{1} s_{2} s_{3}  feasibility c^{T} x/||c|| = 1
1 5 6 2 0 0 feasible 28/7.77
2 6.25 5.5 0 −5 0 infeasible  29/8.04
3 7.5 3 0 0 10 feasible 24/6.66
4 20 0 −2 −60 20 infeasible  40/11.1
5 10 0 −2 0  20 infeasible  20/5.55
6  9  0 0  6  22  feasible  18/4.10
7  0  8 10 20  0  feasible  24/6.66
8  0 12 6  0 −20 infeasible  36/9.98
9  0 18 0 −30 −50 infeasible  54/14.99
10 0  0 18 60 40  feasible  0/0.00

This primal maximization problem can be converted into the dual minimization problem:

min: f_{d}(y_{1}, y_{2}, y_{3}) = 18y_{1} + 60y_{2} + 40y_{3}_{s. t.: \{ \begin{matrix}2y_{1} + 6y_{2} + 2y_{3} ≥ 2 \\ y_{1} + 5y_{2} + 5y_{3} ≥ 3 \\ y_{1} ≥ 0, y_{2} ≥ 0, y_{3} ≥ 0 \end{matrix}}

or

min: f_{d}(y) = b^{T} y
s. t.: \{ \begin{matrix}A^{T} y − c ≥ 0 \\ y ≥ 0 \end{matrix}

The five planes, m = 2 from the constraining inequalities, and n = 3 for the non-negative constraints y_{1} ≤ 0, y_{2} ≤ 0, and y_{3} ≤ 0 for the three variables, form 10 intersection points:

(y_{1}, y_{2}, y_{3}) feasibility objective function
1  (1, 0, 0)  infeasible  18
2  (3, 0, 0)  feasible  54
3 (0, 1/3, 0) infeasible  20
4 (0, 3/5, 0)  feasible  36
5  (0, 0, 1)  feasible  40
6 (0, 0, 3/5) infeasible  24
7 (0, 1/5, 2/5) feasible  28
8 (1/2, 0, 1/2) feasible  29
9 (−2, 1, 0) infeasible  24
10  (0, 0, 0)  infeasible  0

We see that at the feasible point (0, 1/5, 2/5), the dual function f_{d}(y_{1}, y_{2}, y_{3}) = 18y_{1} +60y_{2} +40y_{3} is minimized to be 28, the same as the maximum of the primal function f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} at (5, 6).

Homework:

Develop the code to find all C_{N+M}^{N} intersections formed by N + M given hyper-planes in an N-D space in terms of their equations A_{M×N}x_{N×1} ≤ b_{M×1} and x ≥ 0, identify which of them are vertices of the polytope surrounded by these hype-planes, and find the vertex corresponding to the optimal solution that maximizes f(x) = c^{T} x for a given c.

2.2.5 The Simplex Algorithm

The simplex algorithm finds the optimal solution of a LP problem by an iterative process that traverses along a sequence of edges of the polytopic feasible region, starting at the origin and through a sequence of vertices x with progressively greater objective value f(x), until eventually reaching the optimal solution. By doing so, it avoids checking exhaustively all vertices of the feasible region for optimality.

We first consider the equality constraint Ax = [A_{M×N} | I_{M×M}]x = b of the standard LP problem. This is an under-determined linear system of N + M variables but only M equations, only M of its N + M columns of the coefficient matrix A are independent (assuming rank(A) = M). Initially we choose the M columns of the identity coefficient matrix I as the independent columns, but subsequently we can choose any other M columns of A as the independent columns and convert them into a standard basis vector by Gauss-Jordan elimination together with the corresponding variables in x. The resulting equation Ax = b remains equivalent to the original one, i.e., the M equality constraints are always preserved and never violated in the process.

For convenience of discussion and without loss of generality, we could reorder (not actually carried out) the N + M columns in A, together with the corresponding variables in x, so that the constraining equation would always takes the following form:

Ax = [A_{n} I] [ \begin{matrix}x_{n} \\ x_{b} \end{matrix} ] = A_{n}x_{n} + I x_{b} = b  (2.218)

Now the variables in the M-D vector x_{b} corresponding to the M independent columns in I are the basic variables, while the N-D vector x_{n} corresponding to the remaining N dependent columns, now denoted by A_{n}, are the non-basic variables. This equation will always hold if x_{n} = 0 and x_{b} = b. Such a solution is called a basic solution of the linear system Ax = b:

x = [ \begin{matrix}x_{n} \\ x_{b} \end{matrix} ] = [ \begin{matrix}0 \\ b \end{matrix} ]  (2.219)

Initially, A_{n} is the coefficient matrix in the inequality constraint Ax ≤ b, and the corresponding non-basic variables x_{n} = 0 is actually the origin. But through the iteration, we will keep converting some other columns of A into standard basis vectors {e_{1}, ⋯, e_{M}}, the column vectors of I, by Gauss-Jordan elimination. The variables in x corresponding to the new standard basis vectors become the basic variables, while those corresponding to columns that become non-standard basis vectors become non-basis variables.

The basic solutions corresponding the C_{N+M}^{N} ways to choose M of the N + M columns of A as independent are actually the intersections formed by any N of the N + M constraining hyper-planes in the N-D space. As noted before, only a subset of these C_{N+M}^{N} basic solutions satisfy the M constraints, and they are called basic feasible solutions (BFS). The goal of the iterative process of the simplex algorithm is to find the optimal basic feasible solution that maximizes f(x) without exhausting all C_{N+M}^{N} possibilities. This is done by selecting the columns in such a way that the value of the objective function will always be maximally increased, until eventually we find the optimal solution x^{∗} at one of the vertex of the polytopic feasible region.

Specially, the implementation of the simplex algorithm is based on a tableau with N + M + 1 columns and M + 1 rows, initialized as below:

• The first M rows are for the coefficients in the equality constraints ∑_{j=1}^{N} a_{ij}x_{j}+ s_{i} = b_{i} (i = 1, ⋯, M). An additional (M + 1)st row at the bottom is for the coefficients in the objective function f(x) = ∑_{j=1}^{N} c_{j}x_{j}, which is zero initially at the origin x_{1} = ⋯ = x_{N} = 0, i.e., z − c_{1}x_{1} − ⋯ − c_{N}x_{N} = 0.

• The first N + M columns contain the coefficients for the N original variables {x_{1}, ⋯, x_{N}} and the M slack variables {s_{1}, ⋯, s_{M}}, and an additional (N + M + 1)st column is for the constants b = [b_{1}, ⋯, b_{M}]^{T} on the righthand side of the M constraint equations. The last element of the column is the right-hand side of the objective equation z = f(x), which is initially zero. basic variables x_{1} x_{2}

⋯

x_{N} s_{1} s_{2}

⋯

s_{M}

basic solution

s_{1}  a_{11} a_{12} ⋯ a_{1N} 1 0 ⋯ 0  b_{1}
s_{2}  a_{21} a_{22} ⋯ a_{2N} 0 1 ⋯ 0  b_{2}
⋮  ⋮  ⋮ ...  ⋮  ⋮ ⋮ ... ⋮  ⋮
s_{M}  a_{M1} a_{M2} ⋯ a_{MN} 0 0 ⋯ 1  b_{M}
z  −c_{1} −c_{2} ⋯ −c_{M} 0 0 ⋯ 0  0

(2.220)

At this initial stage of the iteration, the N original variables x_{n} = [x_{1}, ⋯, x_{N}]^{T} are the non-basic variables, while the M slack variables x_{b} = [s_{1}, ⋯, s_{M}]^{T} are the basic variables and their coefficients form an M × M identity matrix I, composed of the standard basis vectors. The corresponding feasible basic solution is simply

x_{b} = [s_{1}, ⋯, s_{M}]^{T} = b, x_{n} = [x_{1}, ⋯, x_{N}]^{T} = 0  (2.221)

that satisfies all the constraints Ax = A_{n}x_{n} + Ix_{b} = b.

In each of the subsequent iterations, we will select one of the non-basic variables, called the entering variable, to replace one of the basic variables, called the leaving variable, in such a way that the value of the objective function value z = f(x) in the last row will be maximally increased, while all constraints remain satisfied. Here are the steps in each iteration:

• Selection of the entering variable

This section is based on the maximization of z = f(x). We select x_{j} in the jth column of the tableau if −c_{j} is most negative, i.e., x_{j} is most heavily weighted by c_{j} = max{c_{1}, ⋯, c_{N}}, so that it will increase z = ∑_{j=1}^{N} c_{j}x_{j} more than any other x_{k} (k ≠ j).

• Selection of the leaving variable

This selection is based on the constraints imposed on the selected entering variable x_{j}. In general, the restriction on x_{j} set by the kth constraint ∑_{j=1}^{N} a_{kj}x_{j} ≤ b_{k} is x_{j} ≤ b_{k}/a_{kj} when a_{kj} > 0. If b_{i}/a_{ij} = min{b_{k}/a_{1j}, ⋯, b_{k}/a_{Mj}}, then the ith constraint is most restrictive on x_{j}, we will therefore select the corresponding basic variable s_{i} as the leaving variable, i.e., it becomes a non-basic variable to be set to zero, so that x_{j} can be maximally increased without violating the constraints. If all a_{kj} < 0 for all k = 1, ⋯, M, variable x_{j} is not bounded.

• Gauss-Jordan elimination based on Pivoting

To convert the entering variable x_{j} to a basic variable to replace the leaving variable s_{i}, we need to turn the corresponding jth column into a standard basis vector e_{i}. This is realized by pivoting on a_{ij} in the following steps: 1.Divide the pivot row r_{i} by a_{ij}: r_{i} ← r_{i}/a_{ij}. Now a_{ij} = 1. 2.Subtract a_{kj}r_{i} from the kth row: r_{k} ← r_{k} − a_{kj}r_{i} so that a_{kj} = 0 for all k = 1, ⋯, M + 1, k ≠ i, including the last row r_{M+1}.

Now the jth column corresponding to the entering variable x_{j} becomes a standard basis vector e_{i}, i.e., x_{j} becomes a basic variable that takes the value of b_{i}, and the column corresponding to the leaving variable s_{i} is no longer a standard basis vector, and s_{i} = 0 as it is now a non-basic variable.

As Gauss-Jordan elimination converts the linear equations to a set of equivalent equations, the constraints remain satisfied through out the process. Although the membership of the basic and non-basic variable groups keeps changing in the iteration, so long as x_{b} = b and x_{n} = 0, the M constraint equations Ax = A_{n}x_{n} + Ix_{b} = b always holds.

This iterative process keeps replacing the slack variables {s_{1}, ⋯, s_{M}} (the basic variables initially) by the original variables {x_{1}, ⋯, x_{N}} (the non-basic variables initially), one at a time, until all original variables become basic variables and all elements in the last row are non-negative.

The final result can be read out directly from the tableau. The variables corresponding to the M standard basis vectors e_{i} (i = 1, ⋯, M) are the final basic variables that take the values in b in the right-most column of the tableau. They form the optimal basic solution, with the maximum of the objective function z = f(x) given by the last element also in the last column. The remaining n variables corresponding to non-standard columns are non-basic variables that take the value zero. When M > N (more constraints than variables), some of the slack variables may remain in the basic variable group taking non-zero values; when M < N, some of the original variable may be in the non-basic group taking the value zero.

Example:

We re-solve the LP problem considered in the previous examples, now in the standard form:

\begin{matrix}max: & f(x) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} + s_{1} = 18 \\ 6x_{1} + 5x_{2} + s_{2} = 60 \\ 2x_{1} + 5x_{2} + s_{3} = 40 \\ x_{1} ≥ 0, x_{2} ≥ 0, s_{1} ≥ 0, s_{2} ≥ 0, s_{3} ≥ 0 \end{matrix} \end{matrix}

The standard form is further converted to a tableau as shown below. The leftmost column indicates the basic variables, the next N = 2 columns are for the variables x and y,the next M = 3 columns are for the slack variables u, v, and w, the right most column is for the constants b_{1}, b_{2}, and b_{3} on the right-hand side of the equations.

x_{1} x_{2} s_{1} s_{2} s_{3} b
s_{1} 2 1 1 0 0 18
s_{2} 6 5 0 1 0 60
s_{3} 2 5 0 0 1 40
z −2 −3 0 0 0 0

In this initial state, the basic variables are s_{1} = 18, s_{2} = 60, and s_{3} = 40, the non-basic variables are x_{1} = x_{2} = 0. The corresponding basic feasible solution is at the origin.

• Select column j = 2 as the pivot column, as −c_{2} = −3 < −c_{1} is the most negative coefficient.

• Select r_{3} as the pivot row, as b_{3}/a_{32} = 40/5 = 8 = min{18/1 = 18, 60/5 = 12, 40/5 = 8}.

• Divide pivot row r_{i} = r_{3} = [2, 5, 0, 0, 1, 40] by the pivot element a_{ij} = 32 = 5 to get r_{i} = r_{3} = [0.4, 1, 0, 0, 0.2, 8].

• Subtract a_{kj}r_{k} = a_{k2}r_{k} from row r_{k}, so that a_{kj} = a_{k2} = 0, (k = 1, ⋯, M + 1 = 4, k ≠ i), including the last row r_{4}.

x_{1} x_{2} s_{1} s_{2} s_{3} b
s_{1} 1.6 0 1 0 −0.2 10
s_{2} 4  0 0 1 −1 20
x_{2} 0.4 1.0 0 0 0.2 8
z −0.8 0 0 0 0.6 24

The entering variable x_{j} = x_{2} becomes a basic variable, and the leaving variable s_{i} = s_{3} becomes a non-basic variable, replaced by new basic variable x_{2}. The corresponding basic feasible solution is at x_{1} = 0, x_{2} = b_{3} = 8, with s_{1} = 10, s_{2} = 20, s_{3} = 0. The objective function value is z = c^{T} x = 24.

• Select column j = 1 as the pivot column, as −c_{1} = −0.8 is the most (only) negative coefficient in the last row.

• Select row i = 2 as the pivot row, as the ratio b_{2}/a_{21} = 20/4 = 5 = min{10/1.6 = 6.25, 20/4 = 5, 8/0.4 = 20}.

• Divide pivot row r_{i} = r_{2} = [4, 0, 0, 1, −1, 20] by the pivot element a_{ij} = a_{21} = 4 to get r_{i} = r_{2} = [1, 0, 0, 0.25, −0.25, 5].

• Subtract a_{kj}r_{k} = a_{k1}r_{k} from row r_{k}, so that a_{kj} = a_{k1} = 0, (k = 1, ⋯, M + 1 = 4), including the last row r_{4}.

x_{1} x_{2} s_{1} s_{2}  s_{3}  b
s_{1} 0 0 1 −0.4 0.2 2
x_{1} 1 0 0 0.25 −0.25 5
x_{2} 0 1 0 −0, 1 0.3 6
z 0 0 0 0.2  0.4 28

The entering variable x_{1} becomes a basic variable, and the leaving variable s_{1} becomes a non-basic variable, replaced by the new basic variable x_{1}. The corresponding basic feasible solution is at x_{1} = b_{2} = 5, x_{2} = b_{3} = 6, with s_{1} = 2, s_{2} = s_{3} = 0. The objective function value is z = c^{T} x = 28. Now that both x_{1} and x_{2} have beome basic variables, the optimal solution has been obtained.

This problem has C_{N+M}^{N} = C_{5}^{2} = 10 basic solutions, corresponding to the same number of intersections, out of which five are feasible, as previously obtained. The simplex method finds three of these feasible solutions, starting from x_{1} = x_{2} = 0 at the origin with z = 0, through the vertex at x_{1} = 0 and x_{2} = 8 with z = 24, to the optimal solution at x_{1} = 5 and x_{2} = 6 with z = 28.

The Matlab code for implementing the simplex algorithm is listed below. The function takes input including matrix A as well as the vectors b and c, and generates the optimal solution x^{∗}, the corresponding maximum value of the ojbective function f(x) = c^{T} x^{∗}, together with the values of the slack vaerible s.

function [x s z] = MySimplex(A,b,c)
[m n]=size(A)                       % coefficient matrix for variables x
x=zeros(n,1);                       % initial zero values for x
s=zeros(m,1);
A=[A eye(m) b; -c zeros(1,m+1)];    % initialization of tableau
[cmin, pc]=min( A(m+1,1:n+m) );     % pc, index of the first pivot column
it=0;
while cmin<0                        % iteration until all c's in last row are zero
it=it+1;
rmin=9e9;
for i=1:m                       % find pivot row
    if A(i,n+m+1)~=0 & A(i,pc)>0
        w=A(i,n+m+1)./A(i,pc);
        if w<rmin
            pr=i;               % pr, index of the pivot row
            rmin=w;
        end
    end
end
p=A(pr,pc);                     % get the pivot element
A(pr,:)=A(pr,:)/p;              % modify pivot row
for i=1:m+1                     % modify all k+1 rows
    if i~=pr
        A(i,:)=A(i,:)-A(pr,:)*A(i,pc);
    end
end
z=A(m+1,n+m+1);                 % maximum value of objective function so far
[cmin, pc]=min( A(m+1,1:n+m) ); % pivot column of next iteration
end
end

2.2.6 Quadratic Programming (QP)

A quadratic programming (QP) problem is to minimize a quadratic function subject to some equality and/or inequality constraints:

min: f(x) = \frac{1}{2}[x − m]^{T} Q[x − m] = \frac{1}{2}x^{T} Qx + c^{T} x + c
s. t.: Ax ≤ b

(2.222)

where Q = Q^{T} is an N × N positive definite matrix, A an M × N matrix, and b is an M-D vector. Also, c = −Qm, c = m^{T} Qm/2. Here the scalar constant c can be dropped as it does not play any role in the optimization. Note that x^{T} Qx is a hyper elliptic paraboloid with the minimum at point m in the N-D space.

We first consider the special case where the QP problem is only subject to equality constraints and we assume m ≤ N, i.e., the number of constraints is smaller than the number of unknowns in x. Then the solution x has to satisfy Ax − b = 0, i.e., it has to be on M hyper planes in the N-D space.

The Lagrangian function of the QP problem is:

L(x, λ) = f(x) + λ^{T} (Ax − b) = \frac{1}{2}x^{T} Qx + c^{T} x + λ^{T} (Ax − b)  (2.223)

To find the optimal solution, we first equate the derivatives of the Lagrangian with respect to both x and λ to zero:

▽_{x}L(x, λ) = Qx + c + A^{T} λ = 0
▽_{λ}L(x, λ) = Ax − b = 0  (2.224)

These two equations can be combined and expressed in matrix form as:

[ \begin{matrix}Q & A^{T} \\ A & 0 \end{matrix} ] [ \begin{matrix}x \\ λ \end{matrix} ] = [ \begin{matrix}−c \\ b \end{matrix} ]  (2.225)

We then solve this system of m + N equations to get the optimal solution x^{∗} and the corresponding λ^{∗}.

Example

min: f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} = \frac{1}{2}[x_{1}, x_{2}] [ \begin{matrix}2 & 0 \\ 0 & 2 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = \frac{1}{2}x^{T} Qx
s. t.: Ax = [1, 1] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = x_{1} + x_{2} = b = 1

where

Q = [ \begin{matrix}2 & 0 \\ 0 & 2 \end{matrix} ] , A = [1, 1], c = [ \begin{matrix}0 \\ 0 \end{matrix} ] , b = 1;

[ \begin{matrix}2 & 0 & 1 \\ 0 & 2 & 1 \\ 1 & 1 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ λ \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 1 \end{matrix} ]

Solving this equation we get the solution x_{1}^{∗} = x_{2}^{∗} = 0.5 and λ^{∗} = −1, at which the function f(x_{1}^{∗}, x_{2}^{∗}) = 0.5 is minimized subject to x_{1} + x_{2} = 1.

If M = N, i.e., the number of equality constraints is the same as the number of variables, then the variable x is uniquely determined by the linear system Ax = b, as the intersect of M = N hyper planes, independent of the objective function f(x). Further if M > N, i.e., the system Ax = b is over constrained, and its solution does not exist in general. It is therefore more interesting to consider QP problems subject to both equality and inequality constraints:

min:  f(x) = \frac{1}{2}x^{T} Qx + c^{T} x
s. t.:  Ax ≤ b

2.2.7 Interior Point Methods

Consider the a general constrained optimization problem:

min: f(x)
s. t.: \{ \begin{matrix}h(x) = 0 \\ g(x) ≤ 0 or g(x) ≥ 0 \\ x ≥ 0 \end{matrix}

(2.226)

By introducing slack variables s ≥ 0, all inequality constraints can be converted into equality constraints:

g(x) ≤ 0 ⟹ g(x) + s = 0,  g(x) ≥ 0 ⟹ g(x) − s = 0  (2.227)

which can then be combined with the equality constraints and still denoted by h(x) = 0, while the slack variables s are also combined with the original variables and still denoted by x. We assume there are in total M equality constraints and N variables. Now the optimization problem can be reformulated as:

min: f(x)
s. t.: \{ \begin{matrix}h(x) = 0 \\ x ≥ 0 \end{matrix} (2.228)

The Lagrangian is

L(x, λ, μ) = f(x) + λ^{T} h(x) − μ^{T} x  (2.229)

The KKT conditions are

\{ \begin{matrix}▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = 0 & (stationarity) \\ h(x) = 0, x ≥ 0 & (primal feasibility) \\ μ ≥ 0 & (dual feasibility) \\ μ_{j}x_{j} = 0, (j = 1, ⋯, N) or XM1 = 0 & (complementarity) \end{matrix} (2.230)

where

X = [ \begin{matrix}x_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & x_{n} \end{matrix} ]_{N×N} , M = [ \begin{matrix}μ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & μ_{N} \end{matrix} ]_{N×N}  (2.231)

and J_{f} (x) is the Jacobian matrix of h(x):

J_{h}(x) = [ \begin{matrix}\frac{∂h_{1}}{∂x_{1}} & ⋯ & \frac{∂h_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂h_{m}}{∂x_{1}} & ⋯ & \frac{∂h_{m}}{∂x_{n}} \end{matrix} ]_{M×N}  (2.232)

To simplify the problem, the non-negativity constraints can be removed by introducing an indicator function:

I(x) = \{ \begin{matrix}0 & x ≥ 0 \\ ∞ & x < 0 \end{matrix}  (2.233)

which, when used as a cost function, penalizes any violation of the non-negativity constraint x ≥ 0. Now the optimization problem can be reformulated as shown below without the non-negative constants x ≥ 0:

min: f(x) + ∑_{i=1}^{N} I(x_{i})
s. t.: h(x) = 0

(2.234)

However, as the indicator function is not smooth and therefore not differentiable, it is approximated by the logarithmic barrier function which approaches I(x) when the parameter t > 0 approaches infinity:

−\frac{1}{t} ln(x) ⟹^{t→∞} I(x)  (2.235)

Now the optimization problem can be written as:

min: f(x) − \frac{1}{t} ∑_{i=1}^{N} ln x_{i}
s. t.: h(x) = 0

(2.236)

The Lagrangian is:

L(x, λ, μ) = f(x) + λ^{T} h(x) − \frac{1}{t} ∑_{j=1}^{N} ln x_{j}  (2.237)

and its gradient is:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − \frac{1}{t}X^{−1}1  (2.238)

We define

μ = \frac{x}{t} = \frac{1}{t}X^{−1}1, i.e. Xμ = XM1 = \frac{1}{t}  (2.239)

and combine this equation with the KKT conditions of the new optimization problem to get

\{ \begin{matrix}g_{f}(x) + J_{h}^{T}(x)λ − μ = 0 & (stationarity) \\ h(x) = 0 & (primal feasibility) \\ XM1 − 1/t = 0 & (complementarity) \end{matrix}  (2.240)

The third condition is actually from the definition of μ given above, but here it is labeled as complementarity, so that these conditions can be compared with the KKT conditions of the original problem. We see that the two sets of KKT conditions are similar to each other, with the following differences:

• The non-negativity x ≥ 0 in the primal feasibility of the original KKT is dropped;

• Consequently the inequality μ ≥ 0 in the dual feasibility of the original KKT conditions is also dropped;

• There is an extra term 1/t in complementarity which will vanish when t → ∞.

The optimal solution that minimizes the objective function f(x) can be found by solving the simultaneous equations in the modified KKT conditions (with no inequalities) given above. To do so, we first express the equations in the KKT conditions above as a nonlinear equation system F(x, λ, μ) = 0, and find its Jacobian matrix J_{F} composed of the partial derivatives of the function F(x, λ, μ) with respect to each of the three variables x, λ, and μ:

F(x, λ, μ) = [ \begin{matrix}g_{f}(x) + J_{h}^{T}(x)λ − μ \\ h(x) \\ XM1 − 1/t \end{matrix} ] , J_{F} = [ \begin{matrix}W(x) & J_{h}^{T}(x) & −I \\ J_{h}(x) & 0 & 0 \\ M & 0 & X \end{matrix} ]

(2.241)

where

W(x) = ▽_{x} [g_{f}(x) + J_{h}^{T}(x)λ − μ]
= ▽_{x}g_{f}(x) − ▽_{x} [∑_{i=1}^{M} λ_{i} \frac{∂h_{i}}{∂x_{1}} , ⋯, ∑_{i=1}^{M} λ_{i} \frac{∂h_{i}}{∂x_{N}} ]^{T} = H_{f}(x) − ∑_{i=1}^{M} λ_{i}H_{h_{i}}(x) (2.242)

is symmetric as the Hessian matrices H_{f} and H_{h_{i}}, (i = 1, ⋯, M) are symmetric.

We can now use the Newton-Raphson method to find the solution of the nonlinear equation F(x, λ, μ) = 0 iteratively:

[ \begin{matrix}x_{n+1} \\ λ & n+1 \\ μ & n+1 \end{matrix} ] = [ \begin{matrix}x_{n} \\ λ & n \\ μ & n \end{matrix} ] + α [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ]  (2.243)

where α is a parameter that controls the step size and [δx_{n}, δλ_{n}, δμ_{n}]^{T} is the search direction (Newton direction), which can be found by solving the following equation:

J_{F}(x_{n}, λ_{n}, μ_{n}) [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ] = [ \begin{matrix}W(x_{n}) & J_{h}^{T}(x_{n}) & −I \\ J_{h}(x_{n}) & 0 & 0 \\ M_{n} & 0 & X_{n} \end{matrix} ] [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ]

= −F(x_{n}, λ_{n}, μ_{n}) = − [ \begin{matrix}g_{f}(x_{n}) + J_{h}^{T}(x_{n})λ_{n} − μ_{n} \\ h(x_{n}) \\ X_{n}M_{n}1 − 1/t \end{matrix} ]  (2.244)

To get the initial values for the iteration, we first find any point inside the feasible region x_{0} and then μ_{0} = x_{0}/t, based on which we further find λ_{0} by solving the first equation in the KKT conditions in Eq. (2.240).

Actually this equation system above can be separated into two subsystems, which are easier to solve. Pre-multiplying X^{−1} to the third equation:

Mδx + Xδμ = −XM1 + 1/t  (2.245)

we get

X^{−1}Mδx + δμ = −M1 + X^{−1}1/t = −μ + X^{−1}1/t  (2.246)

Adding this to the first equation:

Wδx + J_{h}^{T}(x)δλ − δμ = −g_{f}(x) − J_{h}^{T}(x)λ + μ  (2.247)

we get

(W + X^{−1}M)δx + J_{h}^{T}(x)δλ = −g_{f}(x) − J_{h}^{T}(x)λ + X^{−1}1/t = − ▽_{x} L(x, λ, μ)

(2.248)

Combining this equation with the second one, we get an equation system of two vector variables δx and δλ with symmetric coefficient matrix:

[ \begin{matrix}W + X^{−1}M & J_{h}^{T}(x) \\ J_{h}(x) & 0 \end{matrix} ] [ \begin{matrix}δx \\ δλ \end{matrix} ] = − [ \begin{matrix}▽_{x}L(x, λ, μ) \\ h(x) \end{matrix} ]  (2.249)

Solving this equation system we get δx and δλ, and we can further find δμ by solving the third equation

Mδx + Xδμ = −XM1 − 1/t  (2.250)

to get

δμ = X^{−1}1/t − X^{−1}Mδx − μ  (2.251)

We now consider the two special cases of linear and quadratic programming:

• Linear programming:

min: f(x) = c^{T} x
s. t.: h(x) = Ax − b = 0, x ≥ 0

(2.252)

We have

– g_{f}(x) = ▽_{x}f(x) = c – J_{h}(x) = ▽_{x}(Ax − b) = A.

– W(x) = ▽_{x}^{2}L(x, λ, μ) = 0. The Lagrangian is

L(x, λ, μ) = c^{T} x + λ^{T} (Ax − b) − μ^{T} x  (2.253)

The KKT conditions are:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = c + A^{T} λ − μ = 0

h(x) = Ax − b = 0
XM1 − 1/t = 0

The search direction of the iteration can be found by solving the this equation system:

[ \begin{matrix}0 & A^{T} & −I \\ A & 0 & 0 \\ M & 0 & X \end{matrix} ] [ \begin{matrix}δx \\ δλ \\ δμ \end{matrix} ] = − [ \begin{matrix}c + A^{T} λ − μ \\ Ax − b \\ XM1 − 1/t \end{matrix} ]  (2.254)

By solving the equation c + A^{T} λ − μ = 0 we get the initial value λ. The Matlab code for the interior point method for the LP problem is listed below:

function x=InterierPointLP(A,c,b,x)
% given A,b,c of the LP problem and inital value of x,
% find optimal solution x that minimizes c'*x
[m n]=size(A);      % m constraints, n variables
z1=zeros(n);
z2=zeros(m);        % zero matrices in coefficient matrix
z3=zeros(m,n);
I=eye(n);           % identity matrix in coefficient matrix
y=ones(n,1);
t=9;                % initinal value for parameter t
alpha=0.3;          % small enough not to exceed boundary
mu=x./t;
lambda=pinv(A')*(mu-c); % initial value of lambda
w=[x; lambda; mu];      % initial values for all three
B=[c+A'*lambda-mu; A*x-b; x.*mu-y/t];
p=[x(1) x(2)];      % initial guess of solution
while norm(B)>10^(-7)
    t=t*9;          % increase parameter t
    X=diag(x);
    M=diag(mu);
    C=[z1 A' -I; A z2 z3; M z3' X];
    B=[c+A'*lambda-mu; A*x-b; x.*mu-y/t];
    dw=-inv(C)*B;   % find search direction
    w=w+alpha*dw;   % step forward
    x=w(1:n);

        lambda=w(n+1:n+m);
        mu=w(n+m+1:length(w));
        p=[p; x(1), x(2)];
    end
    scatter(p(:,1),p(:,2));  % plot trajectory of solution
end

Example:

A given linear programming problem shown below is first converted into the standard form:

\begin{matrix}max: & f(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix} \end{matrix}  ⟹  \begin{matrix}min: & f(x_{1}, x_{2}, x_{3}, x_{4}, x_{5}) = −2x_{1} − 3x_{2} \\ s. t,: & \{ \begin{matrix}2x_{1} + x_{2} + x_{3} = 18 \\ 6x_{1} + 5x_{2} + x_{4} = 60 \\ 2x_{1} + 5x_{2} + x_{5} = 40 \\ x_{i} ≥ 0, (i = 1, ⋯, 5) \end{matrix} \end{matrix}

or in matrix form:

min: f(x) = c^{T} x
s. t. : h(x) = Ax − b = 0, x ≥ 0

with

x = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \\ x_{5} \end{matrix} ] , c = [ \begin{matrix}−2 \\ −3 \\ 0 \\ 0 \\ 0 \end{matrix} ] , A = [ \begin{matrix}2 & 1 & 1 & 0 & 0 \\ 6 & 5 & 0 & 1 & 0 \\ 2 & 5 & 0 & 0 & 1 \end{matrix} ] , b = [ \begin{matrix}18 \\ 60 \\ 40 \end{matrix} ]

where x_{3}, x_{4}, x_{5} are the slack variables.

We choose an initial value x_{0} = [1, 2, 1, 1, 1]^{T}, and an initial parameter t = 9, which is scaled up by a factor of 9 in each iteration. We also used full Newton step size of α = 1. After 8 iterations, the algorithm converged to the optimal solution x^{∗} = [5, 6]^{T} corresponding to the maximum function value f(x^{∗}) = 2x_{1} + 3x_{2} = 28:

(x_{1}  x_{2})  f(x)  error
1 (1.000000e + 00 2.000000e + 00) −8.000000 52.413951
2 (4.654514e + 00 6.346354e + 00) −28.348090 3.080204
3 (5.040828e + 00 5.946973e + 00) −27.922575 0.213509
4 (4.997282e + 00 6.001764e + 00) −27.999856 0.004262
5 (4.999906e + 00 5.999962e + 00) −27.999697 0.000303
6 (4.999989e + 00 5.999996e + 00) −27.999966 0.000034
7 (4.999999e + 00 6.000000e + 00) −27.999996 0.000004
8 (5.000000e + 00 6.000000e + 00) −28.000000 0.000000

At the end of the iteration, we also get the values of the slack variables: x_{3} = 2, x_{4} = x_{5} = 0.

• Quadratic programming:

min: f(x) = \frac{1}{2}x^{T} Qx + c^{T} x
s. t.: h(x) = Ax − b = 0, x ≥ 0

(2.255)

We have

– g_{f}(x) = ▽_{x}f(x) = Qx + c

– J_{h}(x) = ▽_{x}(Ax − b) = A.

– W(x) = ▽_{x}^{2}L(x, λ, μ) = Q. The Lagrangian is

L(x, λ, μ) = \frac{1}{2}x^{T} Qx + c^{T} x + λ^{T} (Ax − b) − μ^{T} x  (2.256)

The KKT conditions are:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = Qx + c + A^{T} λ − μ = 0

h(x) = Ax − b = 0
XM1 − 1/t = 0

The equation system for the Newton-Raphson method is:

[ \begin{matrix}Q & A^{T} & −I \\ A & 0 & 0 \\ M & 0 & X \end{matrix} ] [ \begin{matrix}δx \\ δλ \\ δμ \end{matrix} ] = − [ \begin{matrix}Qx + c + A^{T} λ − μ \\ Ax − b \\ XM1 − 1/t \end{matrix} ]  (2.257)

The Matlab code for the interior point method for the QP problem is listed below:

function [x mu lambda]=InteriorPointQP(Q,A,c,b,x)
n=length(c);        % n variables
m=length(b);     % m constraints
z2=zeros(m);
z3=zeros(m,n);
I=eye(n);
y=ones(n,1);
t=9;                % initinal value for parameter t
alpha=0.1;        % stepsize, small enough not to exceed boundary
mu=x./t;
lambda=pinv(A')*(mu-c-Q*x); % initial value of lambda
w=[x; lambda; mu];    % initial value
B=[Q*x+c+A'*lambda-mu; A*x-b; x.*mu-y/t];
while norm(B)>10^(-7)
    t=t*9;         % increase parameter t
    X=diag(x);
    M=diag(mu);
    C=[Q A' -I; A z2 z3; M z3' X];
    B=[Q*x+c+A'*lambda-mu; A*x-b; x.*mu-y/t];
    dw=-inv(C)*B;   % find search direction

        w=w+alpha*dw;   % step forward
        x=w(1:n);
        lambda=w(n+1:n+m);
        mu=w(n+m+1:length(w));
    end
end

Example:

A 2-D quadratic programming problem shown below is to minimize four different quadratic functions under the same linear constraints as the liniear programming problem in the previous example. In general, the quadratic function is given in the matrix form:

min: f(x) = [x − m]^{T} Q[x − m] = x^{T} Qx + c^{T} x + d
s. t.: h(x) = Ax − b = 0, x ≥ 0

with following four different sets of function parameters:

Q = [ \begin{matrix}4 & −1 \\ −1 & 4 \end{matrix} ] , [ \begin{matrix}2 & 1 \\ 1 & 2 \end{matrix} ] , [ \begin{matrix}2 & 1 \\ 1 & 2 \end{matrix} ] , [ \begin{matrix}3 & −1 \\ −1 & 3 \end{matrix} ]

m = [ \begin{matrix}4 \\ 10 \end{matrix} ] , [ \begin{matrix}9 \\ 5 \end{matrix} ] , [ \begin{matrix}7 \\ 2 \end{matrix} ] , [ \begin{matrix}−1 \\ 6 \end{matrix} ]

As shown in the four panels in the figure below, the iteration of the interior point algorithm brings the solution from the same initial position at x_{0} = [2, 1]^{T} to the final position for each of the four objective functions f(x), the optimal solution, which is on the boundary of the feasible region in all cases except the third one, where the optimal solution is inside the feasible region, i.e., the optimization problem is not constrained. Also note that the trajectories of the solution during the iteration go straightly from the initial guess to the final optimal solution by following the negative gradient direction of the quadratic function in all cases execpt in the last one, where it makes a turn while approaching the vertical boundary, due obviously to the significantly greater value of the log barrier function close to the boundary.

We also note that the dual problem in the SVM algorithm in Eq. (2.196) is in the same form as in Eq. (2.255), i.e., it is a quadratic programming problem that can be solved by the interior point algorithm.

3 Review of Probability and Statistics

3.1 Univariate and Multivariate Random Variables

3.1.1 Univariate Variables

• Random Experiment and its Sample Space

A random experiment is a procedure that can be repeated an infinite number of times and has a set of possible outcomes. The sample space Ω of a certain random experiment is the totality of all its possible outcomes. Ω may be either countable (discrete) or uncountable (continuous).

For example, “Randomly pick a card from a deck of 10 cards labeled 0, 1, ⋯, 9” is a random experiment. The sample space is a set of the 10 possible outcomes: Ω = {0, 1, ⋯, 9}.

• Random Event and its Probability

A random event A ⊂ Ω is a subset of Ω, which can be a null set (empty set) ∅, a proper subset, e.g., A = {0, 2, 4, 6, 8} ⊂ Ω = {0, 1, ⋯, 9}, or the entire Ω.

Event A occurs if the outcome s ∈ A is a member of A. The probability P (A) of an event A is a real-valued function that maps A to a real number 0 ≤ P (A) ≤ 1. In particular, P (∅) = 0, and P (Ω) = 1.

For example, “The randomly chosen card has a number smaller than 4” is a random event, which is represented by a subset A = {0, 1, 2, 3} ⊂ Ω. The probability of this event A is P (A) = 4/10 = 0.4. Event A occurs if the outcome is one of the member of A (e.g., 2).

• Probability Space

The triple (Ω, A, P) of sample space Ω, events A and probability P is called the probability space.

• Random Variables

A random variable x = x(s) is a real-valued function that maps any outcome s ∈ Ω into a real number x, which can be either continuous or discrete.

• Cumulative and Density Distribution Functions

The cumulative distribution function (cdf) of a random variable x is defined as

F_{x}(ξ) = P (x < ξ)  (3.1)

and we have

P (a ≤ x < b) = F_{x}(b) − F_{x}(a)  (3.2)

The probability density function (pdf) of x is defined as

p_{x}(x) = \frac{d}{dξ} F_{x}(ξ)  (3.3)

and we have

F_{x}(ξ) = ∫_{−∞}^{ξ} p(x)dx, and P (a ≤ x < b) = F_{x}(b)−F_{x}(a) = ∫_{a}^{b} p(x)dx

(3.4)

and

F_{x}(∞) = ∫_{−∞}^{∞} p(x)dx = 1  (3.5)

• Discrete Random Variables

If a random variable x can only take one of a set of finite number of discrete values {x_{i} i = 1, ⋯, n}, then its probability distribution is

P (x = x_{i}) = P (x_{i}) = P_{i} (i = 1, ⋯, n)  (3.6)

and we have

0 ≤ P_{i} ≤ 1,  ∑_{i=1}^{n} P_{i} = 1  (3.7)

The cumulative distribution function is

F_{x}(ξ) = P (x < ξ) = ∑_{x_{i}<ξ} P (x_{i}) = ∑_{i=1}^{k} P_{i}  (3.8)

where we assumed x_{1} < ⋯ < x_{n} and ξ = x_{k+1}.

• Expectation

The expectation (or mean value, average) of a random variable x is defined as the average of all possible values of x weighted by their corresponding probabilities:

μ_{x} = E(x) = ∑_{i=1}^{n} x_{i}P (x_{i}) = ∑_{i=1}^{n} x_{i}P_{i}  (3.9)

if x is discrete, or

μ_{x} = E(x) = ∫_{−∞}^{∞} x p(x)dx  (3.10)

If x is continuous.

When the probability p or P is not available, we can estimate the expectation E(x) based on a set of samples {x_{1}, ⋯, x_{k}} of a discrete random variable x:

μ̂_{x} = x̄ = \frac{1}{k} ∑_{i=1}^{k} x_{i}  (3.11)

Sometimes for convenience the mean of a random variable can be Given a random variable x with mean μ_{x}, we can define another random variable y as

y = x − μ_{x}, E(y) = E(x − μ_{x}) = E(x) − μ_{x} = 0  (3.12)

Variable y has the same variance σ_{y}^{2} = σ_{x}^{2} as x, and can be considered as a translated (shifted) version of x. This conversion is carried out often for mathematical convenience without loss of generality.

• Variance The variance of a random variable x is defined as

σ^{2} = V ar(x) =^{△} E[(x − μ)^{2}] = ∫_{−∞}^{∞}(x − μ)^{2}p(x)dx ≥ 0  (3.13)

if x is continuous, or

σ^{2} = V ar(x) = E[(x − μ)^{2}] = ∑_{i=1}^{n}(x_{i} − μ)^{2}P_{i} ≥ 0  (3.14)

if x is discrete. We note that σ^{2} is non-negative, and σ^{2} = 0 only in the special case when x = μ_{x} is no longer random.

The standard deviation of X is defined as

σ = \sqrt{V ar(x)} = \sqrt{σ^{2}}  (3.15)

We have

σ^{2} = V ar(x) = E[(x − μ)^{2}] = E(x^{2} − 2μx + μ^{2})
= E(x^{2}) − 2μE(x) + μ^{2} = E(x^{2}) − μ^{2}  (3.16)

When the probability p or P is not available, we can estimate the variance σ^{2}(x) based on a set of samples {x_{1}, ⋯, x_{k}} of a discrete random variable x:

σ̂^{2} = \frac{1}{k − 1} ∑_{i=1}^{k}(x_{i} − μ)^{2}  (3.17)

• The Median and Mode

The median of a random variable x is defined as the midle value of its distribution. If x is discrete, then its median x̃ is defined as

x̃ = \{ \begin{matrix}x_{(n+1)/2} & if n is odd \\ \frac{x_{n/2}+x_{n/2+1}}{2} & if n is odd \end{matrix}  (3.18)

If x is continuous, then the median x̃ is defined such that

∫ x̃ ; −∞ p(x)dx = ∫ ∞ ; x̃ p(x)dx = 0.5  (3.19)

The mode: the value that appears most frequetnly among {x_{1}, ⋯, x_{n}}.

x̃ = arg max_{x} p(x), i.e., p(x̃) = max_{x} p(x)  (3.20)

The mean, median, and mode are the same if p(x) is unimodal (e.g., a normal distribution), but they are all different from each other if p(x) is asymmetric.

For example, given n = 8 values in {1, 2, 3, 3, 4, 4, 4, 5}, we can find their mean as

x̄ = \frac{1}{8}(1+2+3+3+4+4+4+5) = 1×\frac{1}{8}+2×\frac{1}{8}+3×\frac{2}{8}+4×\frac{3}{8}+5×\frac{1}{8} = \frac{26}{8} = 3.25

(3.21)

The median and mode are respectively 3.5 and 4.

• Intersection and Union of Events

Let A and B be two events defined over Ω.

– The Union (OR) of events A and B is an event whose outcomes belong to either A or B, A ∪ B.

P (A ∪ B) = P (A) + P (B) − P (A ∩ B)  (3.22)

In particular, if A and B are mutually exclusive, i.e.,

A ∩ B = ∅  (3.23)

then

P (A ∪ B) = P (A) + P (B)  (3.24)

– The Intersection (AND) of events A and B is an event whose outcomes belong to both A and B, A ∩ B.

P (A, B) = P (A ∩ B) = P (B|A)P (A) = P (A|B)P (B)  (3.25)

• Independent Variables Two random variables A and B are independent if and only if

P (A ∩ B) = P (A)P (B)  (3.26)

Similarly, a set of N variables are independent iff

p(x_{1}, ⋯, x_{N}) = p(x_{1}) p(x_{2}) ⋯ p(x_{N})  (3.27)

• Conditional Probability

The conditional probability P (A|B) is the probability of event A given that event B has already occurred. Given P (B) as well as P (A|B), we get

P (A, B) = P (A|B)P (B)  (3.28)

We have

P (A|B) = \frac{P (A, B)}{P (B)} = \frac{P (B|A)P (A))}{P (B)},  (3.29)

We further have

\begin{matrix}P (A|B) P (B|C) = P (B|A) P (A|C) \\ P (A|B, C) P (B|C) = P (B|A, C) P (A|C) \end{matrix} \} = P (A, B|C)  (3.30)

If events A and B are independent, i.e., P (A, B) = P (A)P (B), then

P (A|B) = \frac{P (A, B)}{P (B)} = \frac{P (A)P (B)}{P (B)} = P (A)  (3.31)

• Marginal Probability of A is the probability of event A averaging over information about B (the value of B is unknown):

p(x) = ∫ p(x, y) dy, p(y) = ∫ p(x, y) dx  (3.32)

• Partition Theorem (Law of Total Probability)

Let {A_{1}, ⋯, A_{n}} be a set of mutually exclusive events that partition the sample space Ω, i.e.,

⋃_{i=1}^{n} A_{i} = Ω, and A_{i} ∩ A_{j} = ∅ ∀i ≠ j  (3.33)

then for any event B ⊂ Ω, we have

P (B) = ∑_{i=1}^{n} P (B, A_{i}) = ∑_{i=1}^{n} P (B|A_{i})P (A_{i})  (3.34)

Proof:

P (B) = P (B ∩ Ω) = P (B ∩ (⋃_{i=1}^{n} A_{i})) = P (⋃_{i=1}^{n}(B ∩ A_{i}))

= ∑_{i=1}^{n} P (B ∩ A_{i}) = ∑_{i=1}^{n} P (B|A_{i})P (A_{i})  (3.35)

• Bayes’ Theorem

P (A_{i}|B) = \frac{P (B|A_{i}) P (A_{i})}{P (B)} = \frac{P (B|A_{i}) P (A_{i})}{∑_{i=1}^{n} P (B|A_{i})P (A_{i})}  (3.36)

Based on Bayes’ theorem, Bayesian inference is of essential importance and is widely apllied in many fields, and it plays an essential role in machine learning, as some most important algorithms are based on it.

• Normal (Gaussian) Distribution Random variable X has a normal distribution if its pdf is:

N (x, μ, σ) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(x − μ)^{2}}{2σ^{2}} )  (3.37)

It can be shown that

∫_{−∞}^{∞} N (x, μ, σ)dx = 1  (3.38)

E(x) = ∫_{−∞}^{∞} xN (x, μ, σ)dx = μ  (3.39)

and

V ar(x) = ∫_{−∞}^{∞}(x − μ)^{2}N (x, μ, σ)dx = σ^{2}  (3.40)

• Function of Random Variable

A function y = f(x) of a random variable x with pdf p_{X}(x) is also a random variable. If the function is diferentiable and invertible, then we can find its pdf. We first consider its cdf in the following two cases. First, if y = f(x) is a monotonically increasing function (dy/dx > 0), then we have

F_{Y} (y) = P (Y < y) = P (f(X) < y) = P (X < f^{−1}(y)) = ∫_{−∞}^{x} p_{X}(u)du|_{x=f^{−1}(y)}

(3.41)

and its pdf is:

p_{Y} (y) = \frac{d}{dy} F_{Y} (y) = \frac{d}{dy} ∫_{−∞}^{x} p_{X}(u)du|_{x=f^{−1}(y)} = [p_{X}(x)\frac{dx}{dy}]_{x=f^{−1}(y)}

(3.42)

Next, if y = f(x) is monotonically decreasing (dy/dx < 0), then its cdf is:

F_{Y} (y) = P (Y < y) = P (f(X) < y) = P (X > f^{−1}(y)) = ∫_{x}^{∞} p_{X}(u)du|_{x=f^{−1}(y)}

(3.43)

and its pdf is:

p_{Y} (y) = \frac{d}{dy} F_{Y} (y) = \frac{d}{dy} ∫_{x}^{∞} p_{X}(u)du|_{x=f^{−1}(y)} = [−p_{X}(x)\frac{dx}{dy}]_{x=f^{−1}(y)}

(3.44)

Combining both cases, we get

p_{Y} (y) = [p_{X}(x) |\frac{dx}{dy}|]_{x=f^{−1}(y)}  (3.45)

Alternatively, consider the probability for a ≤ x ≤ b, same as the probability for f(a) ≤ y = f(x) ≤ f(b), i.e.,

∫_{a}^{b} p_{X}(x)dx = ∫_{f(a)}^{f(b)} p_{Y} (y)dy  (3.46)

If the interval b−a is infinitesimally small and so is f(b)−f(a), then p_{X}(x) and p_{Y} (y) become constant, we have

p_{X}(x)dx = p_{Y} (y)dy, i.e., p_{Y} (y) = [p_{X}(x) |\frac{dx}{dy}|]_{x=f^{−1}(y)} (3.47)

where the absolute value |dx/dy| ≥ 0 guarantees that p_{Y} (y) > 0 as well as p_{X}(x) > 0.

3.1.2 Multivariate random vectors

• Multiple Random Variables

An outcome of a random experiment may need to be described by a set of n > 1 random variables {x_{1}, ⋯, x_{n}}, which can be more concisely represented as a random vector x = [x_{1}, ⋯, x_{n}]^{T}. For example, in signal processing x can be a set of time samples x_{i} = x(t_{i}) of a random signal x(t) (a random or stochastic process), and in pattern recognition x can represent a pattern, an object of interest, in terms of its features that describe the pattern.

• Joint Distribution Function and Density Function The joint distribution function of a random vector x is defined as

F_{X}(u_{1}, ⋯, u_{n}) = P (x_{1} < u_{1}, ⋯, x_{n} < u_{n})
= ∫_{−∞}^{u_{1}} ⋯ ∫_{−∞}^{u_{n}} p(ξ_{1}, ⋯, ξ_{n})dξ_{1} ⋯ dξ_{n} (3.48)

where p(ξ_{1}, ⋯, ξ_{n}) is the joint density function of the random vector x in the n-dimensional vector space. In particular, if all these random variables are independent, then p(x_{1}, ⋯, x_{n}) = p(x_{1}) ⋯ p(x_{n}) = ∏_{i=1}^{n} p(x_{i}).

• Mean Vector The expectation or mean of a random variable x_{i} is defined as

μ_{i} = E(x_{i}) = ∫_{−∞}^{∞} ⋯ ∫_{−∞}^{∞} ξ_{i} p(ξ_{1}, ⋯, ξ_{n}) dξ_{1} ⋯ dξ_{n}  (3.49)

The mean vector of a random vector x is defined as

m = E(x) = [E(x_{1}), ⋯, E(x_{n})]^{T} = [μ_{1}, ⋯, μ_{n}]^{T}  (3.50)

which can be interpreted as the center of gravity of an n-dimensional object with p(x_{1}, ⋯, x_{n}) being the density function.

• Covariance Matrix The covariance of random variables x_{i} and x_{j} is defined as

σ_{ij}^{2} = Cov(x_{i}, x_{j}) = E[(x_{i} − μ_{i})(x_{j} − μ_{j})]
= E[x_{i}x_{j}] − E[x_{i}]μ_{j} − μ_{i}E[x_{j}] + μ_{i}μ_{j} = E[x_{i}x_{j}] − μ_{i}μ_{j}
= ∫_{−∞}^{∞} ⋯ ∫_{−∞}^{∞} ξ_{i}ξ_{j} p(ξ_{1}, ⋯, ξ_{n}) dξ_{1} ⋯ dξ_{n} − μ_{i}μ_{j}  (3.51)

Note that σ_{ij}^{2} = σ_{ji}^{2} can be either positive or negative. Specially when i = j, σ_{ii}^{2} = E[(x_{i} − μ_{i})^{2}] = E(x_{i}^{2}) − μ_{i}^{2} ≥ 0 is the variance of variable x_{i}, which can be interpreted as the amount of dynamic energy (or information) contained in x_{i}.

If an off-diagonal component σ_{ij}^{2} = 0 is zero, the two variables x_{i} and x_{j} are uncorrelated or decorrelated, and

E(x_{i}x_{j}) = E(x_{i})E(x_{j}) = μ_{i}μ_{j}  (3.52)

If all off-diagonal components σ_{ij} = 0 are zero, then x is uncorrelated. If x_{i} (i = 1, ⋯, n) are independent (p(x_{1}, ⋯, x_{n}) = p(x_{1}) ⋯ p(x_{n})), then they are also uncorrelated. But uncorrelated variables are not necessarily independent. If x is normally distributed, then its components are independent if and only they are uncorrelated.

The covariance matrix of a random vector x is defined as

Σ_{x} = E[(x − m) (x − m)^{T}] = E(xx^{T}) − mm^{T} = [ \begin{matrix}σ_{11}^{2} & ⋯ & σ_{1n}^{2} \\ ⋮ & ... & ⋮ \\ σ_{n1}^{2} & ⋯ & σ_{nn}^{2} \end{matrix} ] ,

(3.53)

The ith diagonal component σ_{ii}^{2} = E[(x_{i} −μ_{i})^{2}] is the variance of x_{i}, and the trace can be interpreted as the total energy or information contained in x:

tr Σ = ∑_{i=1}^{n} σ_{i}^{2} = ∑_{i=1}^{n} λ_{i}  (3.54)

where λ_{1}, ⋯, λ_{n} are the eigenvalues of Σ.

As σ_{ij}^{2} = σ_{ji}^{2}, Σ = Σ^{T} is symmetric. Also, Σ is positive semi-definite (Section ?? in Appendix A), i.e., the following holds for any z ≠ 0:

z^{T} Σz = z^{T} E[(x − m) (x − m)^{T}]z = E[z^{T} (x − m) (x − m)^{T} z]
= E[(z^{T} (x − m))^{2}] ≥ 0  (3.55)

All eigenvalues of this positive semi-definite matrix are non-negative, and they can be sorted so that λ_{1} ≥ ⋯ ≥ λ_{n} ≥ 0. Consequently, we have

tr Σ = ∑_{i=1}^{N} λ_{i} ≥ 0, det Σ = ∏_{i=1}^{N} λ_{i} ≥ 0  (3.56)

The equality holds only in the special case when x = m, i.e., x is deterministic. If this is not the case, the inequality always holds.

The Pearson correlation coefficient between two random variables x_{i} and x_{j} is defined as the normalized covariance σ_{ij}^{2}, when it is divided by the standard deviations σ_{i} and σ_{j}:

ρ_{ij} = \frac{σ_{ij}^{2}}{\sqrt{σ_{i}^{2}σ_{j}^{2}}} = \frac{σ_{ij}^{2}}{σ_{i}σ_{j}}  (3.57)

As the covariance can be considered as an inner product of x_{i} and x_{j}, σ_{ij}^{2} = ⟨x_{i}, x_{j}⟩, the Cauchy-Schwarz inequality (Section 1 in Appendix A) applies:

|⟨x_{i}, x_{j}⟩|^{2} = σ_{ij}^{4} ≤ ⟨x_{i}, x_{i}⟩⟨x_{j}, x_{j}⟩ = σ_{i}^{2}σ_{j}^{2}, or (\frac{σ_{ij}^{2}}{σ_{i}σ_{j}} )^{2} = ρ^{2} ≤ 1

(3.58)

i.e.,

−1 ≤ ρ ≤ 1  (3.59)

• Sample correlation coefficient

The sample correlation coefficient of two random variables x and y can be found based on a set of samples of the two variables {(x_{i}, y_{i}), i = 1, ⋯, k}:

r_{xy} = \begin{matrix} & σ̂_{xy}^{2} \\ σ̂_{x} & σ̂_{y} \end{matrix} = \frac{∑_{i=1}^{k}(x_{i} − x̄)(y_{i} − ȳ)}{\sqrt{(∑_{i=1}^{k}(x_{i} − x̄)^{2}) (∑_{i=1}^{k}(y_{i} − ȳ)^{2})}}  (3.60)

• Correlation Matrix The correlation matrix of x is defined as

R_{x} = E(xx^{T}) = Σ_{x} + mm^{T} = [ \begin{matrix}r_{11}^{2} & ⋯ & r_{1n}^{2} \\ ⋮ & ... & ⋮ \\ r_{n1}^{2} & ⋯ & r_{nn}^{2} \end{matrix} ]  (3.61)

where

r_{ij} = E(x_{i}x_{j}) = σ_{ij}^{2} + μ_{i}μ_{j} (i, j = 1, ⋯, n)  (3.62)

If m = 0, then Σ = R.

• Mean and Covariance under Orthogonal Transforms

Given an orthogonal matrix A = [a_{1}, ⋯, a_{n}] composed of n orthogonal column vectors satisfying a_{i}^{T} a_{j} = δ_{ij}, i.e., A^{T} = A^{−1}, an orthogonal transform of x can be defined as

y = [ y_{1} ; ⋮ ; y_{n} ] = A^{T} x = [ a_{1}^{T} ; ⋮ ; a_{n}^{T} ] x, i.e. y_{i} = a_{i}^{T} x, (i = 1, ⋯, n)

(3.63)

The resulting y is also a random vector. The inverse transform is:

x = Ay = [a_{1}, ⋯, a_{n}] [ y_{1} ; ⋮ ; y_{n} ] = ∑_{i=1}^{n} y_{i}a_{i}  (3.64)

Consider the squared norm of vector x, the total amount of energy contained in x:

||x||^{2} = x^{T} x = (∑_{i=1}^{n} y_{i}a_{i})^{T} (∑_{j=1}^{n} y_{j}a_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} y_{i}y_{j} a_{i}^{T} a_{j} = ∑_{i=1}^{n} y_{i}^{2} = ||y||^{2}

(3.65)

This is Parseval’s theorem, indicating that the total energy is conserved under any orthogonal transform.

The mean vector m_{y} and the covariance matrix Σ_{y} of y are related to the m_{x} and Σ_{x} of x by:

m_{y} = E(y) = E(A^{T} x) = A^{T} E(x) = A^{T} m_{x}  (3.66)

and

Σ_{y} = E(yy^{T}) − m_{y}m_{y}^{T} = E(A^{T} xx^{T} A) − A^{T} m_{x}m_{x}^{T} A
= A^{T} E(xx^{T})A − A^{T} m_{x}m_{x}^{T} A = A^{T} [E(xx^{T}) − m_{x}m_{x}^{T}]A = A^{T} Σ_{x}A (3.67)

The orthogonal transform does not change the trace of Σ: Due to the commutativity of trace: tr(AB) = tr(BA), we have:

trΣ_{y} = tr(A^{T} Σ_{x}A) = tr(A^{T} AΣ_{x}) = trΣ_{x}  (3.68)

i.e., the total amount of dynamic energy or information contained in x is conserved by the orthogonal transform y = A^{T} x.

• Estimation of m and Σ

When p(x) = p(x_{1}, ⋯, x_{n}) is not known, m and Σ cannot be found based on their definitions. However, they can be estimated if a set of outcomes (samples) x_{i}, (i = 1, ⋯, k) of the random experiment in question can be observed.

The sample mean m̂ can be obtained as

m̂ = \frac{1}{k} ∑_{i=1}^{k} x_{i}  (3.69)

The unbiased sample covariance Σ̂ can be obtained as

Σ̂ = \frac{1}{k − 1} ∑_{i=1}^{k}(x_{i} − m̂)(x_{i} − m̂)^{T} = \frac{1}{k − 1} ∑_{i=1}^{k} (x_{i}x_{i}^{T} − x_{i}m̂^{T} − m̂x_{i}^{T} + m̂m̂^{T} )

= \frac{1}{k − 1} (∑_{i=1}^{k} xx^{T} − km̂m̂^{T} ) = R̂ − \frac{k}{k − 1}m̂m̂^{T}  (3.70)

where

R̂ = \frac{1}{k − 1} ∑_{i=1}^{k} x_{i}x_{i}^{T}  (3.71)

If k ≫ 1, then k − 1 ≈ k and the above can be approximated as

Σ̂ ≈ \frac{1}{k} ∑_{i=1}^{k} x_{i}x_{i}^{T} − m̂m̂^{T} ,  R̂ ≈ \frac{1}{k} ∑_{i=1}^{k} x_{i}x_{i}^{T}  (3.72)

If all k samples in the dataset X are independent, then the rank of the n×n estimated covariance matrix Σ̂ is at most k − 1, due to the constraining equation:

∑_{i=1}^{k}(x_{i} − m̂) = ∑_{i=1}^{k} x_{i} − km̂ = 0  (3.73)

If k−1 < n, then Σ̂ does not have a full rank and is therefore noninvertible, i.e., its inverse does not exist.

• Function of Random Vector

Eq. (3.45) for the pdf p_{Y} (y) of a function y = f(x) of a single variable x can be generalized to the pdf of a functions of a random vector. Given a random vector x = [x_{1}, ⋯, x_{n}]^{T} with pdf p_{x}(x), we can define another random vector y = [y_{1}, ⋯, y_{m}]^{T} of which y_{j} = f_{j}(x_{1}, ⋯, x_{n}) for all j = 1, ⋯, m. This function can be written in vector form y = f(x), or x = f^{−1}(y). Then the pdf of y is

p_{Y} (y) = [p_{X}(x) |det J_{f^{−1}} |]_{x=f^{−1}(y)}  (3.74)

where | det J_{f^{−1}}| is the absolute value of the determinant of the Jacobian matrix:

J_{f^{−1}} = [ \begin{matrix}\frac{∂x_{1}}{∂y_{1}} & ⋯ & \frac{∂x_{1}}{∂y_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂x_{m}}{∂y_{1}} & ⋯ & \frac{∂x_{m}}{∂y_{n}} \end{matrix} ]_{x=f^{−1}(y)}  (3.75)

Specially, for a linear function y = f(x) = Ax, J_{f} = A and J_{f^{−1}} = A^{−1}, and

p_{Y} (y) = [| det A^{−1}|p_{X}(x)]_{x=A^{−1}y}  (3.76)

3.1.3 The Normal Distribution

The normal or Gaussian distribution is the most important pdf widely used to model observed data set drawn from an unknown distribution, due to two of its properties:

• Central limit theorem

The distribution of the sum (average or linear combination) of n independent random variables approaches Gaussian as n → ∞.

For example, the face value of a dice has a uniform distribution from 1 to 6. But the distribution of the sum of a pair of dice is no longer uniform. It has a maximum probability at the mean of 7. As the number of dice increases, the distribution of the sum of the face values will be better approximated by a Gaussian. This is illustrated as in the figure below showing n = 1, 3, 5, 7. We see that when

• Maximization of entropy

Among all possible probability density functions (pdfs) with the same variance, the normal distribution has the maximum entropy or uncertainty.

The proof of this property will be given later.

The univariate normal probability distribution is

p(x) = N (x, μ, σ^{2}) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(x − μ)^{2}}{2σ^{2}} )  (3.77)

In multivariate case, the joint pdf of n normally distributed random variables x = [x_{1}, ⋯, x_{n}]^{T} is

p(x) = N (x, m, Σ) = \frac{1}{(2π)^{n/2} |Σ|^{1/2}} exp [−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)]
= exp (−\frac{1}{2}(n log(2π) + log |Σ|)) exp [−\frac{1}{2}(x^{T} Σ^{−1}x − 2m^{T} Σ^{−1}x + m^{T} Σ^{−1}m)]
= exp (−\frac{1}{2}(n log(2π) + log |Σ| + m^{T} Σ^{−1}m)) exp [−\frac{1}{2}(x^{T} Σ^{−1}x − 2m^{T} Σ^{−1}x)] (3.78)

Here m = E(x) and Σ = Cov(x) are respectively the mean vector and covariance matrix of x. The covariance matrix Σ = Σ^{T} is symmetric, and positive semidefinite. We further define the log-normal distribution as:

ψ(x) = log N (x, m, Σ) = −\frac{1}{2}(x−m)^{T} Σ^{−1}(x−m)−\frac{n}{2} log(2π)−\frac{1}{2} log |Σ| (3.79)

and get its gradient vector and Hessian matrix:

g_{ψ}(x) = \frac{d}{dx}ψ(x) = −Σ^{−1}(x − m)  (3.80)
H_{ψ}(x) = \frac{d^{2}}{dx^{2}} ψ(x) = −Σ^{−1}  (3.81)

From these expressions we can find the mean m at which N (m, Σ) reaches maximum can be found by solving the equation

g_{ψ}(x) = −Σ^{−1}(x − m) = 0  (3.82)

to get x = m, and the covariance Σ can be directly found from the second equation:

Σ = −H_{ψ}^{−1}(x)  (3.83)

The shape of this normal distribution in the n-dimensional space can be represented by the iso-hypersurface in the space determined by equation

N (x, m, Σ) = c_{0}  (3.84)

where c_{0} is a constant. Or, equivalently, this equation can be written as

(x − m)^{T} Σ^{−1}(x − m) = c_{1}  (3.85)

where c_{1} is another constant related to c_{0}, m and Σ.

In particular, when n = 2, x = [x_{1}, x_{2}]^{T}, and we assume

Σ^{−1} = [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ]  (3.86)

then the equation above becomes

(x − m)^{T} Σ^{−1}(x − m) = [x_{1} − μ_{1}, x_{2} − μ_{2}] [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ] [ \begin{matrix}x_{1} − μ_{1} \\ x_{2} − μ_{2} \end{matrix} ]

= A(x_{1} − μ_{1})^{2} + B(x_{1} − μ_{1})(x_{2} − μ_{2}) + C(x_{2} − μ_{2})^{2}
= Ax_{1}^{2} + Bx_{1}x_{2} + Cx_{2}^{2} − (2Aμ_{1} + Bμ_{2})x_{1} − (2Cμ_{2} + Bμ_{1})x_{2} + (Aμ_{1}^{2} + Bμ_{1}μ_{2} + Cμ_{2}^{2}) = c_{1} (3.87)

As Σ is positive semi-definite, and so is Σ^{−1}, we have

|Σ^{−1}| = AC − B^{2}/4 ≥ 0, i.e. Δ = B^{2} − 4AC < 0  (3.88)

i.e., the quadratic equation above represents an ellipse (instead of other quadratic curves such as hyperbola and parabola) centered at m = [μ_{1}, μ_{2}]^{T}.

Recall in general that the discriminant Δ = B^{2} −4AC of a quadratic equation

f(x, y) = Ax^{2} + Bxy + Cy^{2} + Dx + Ey + F = 0  (3.89)

determines the shpae of the corresponding conic section in a plane:

f(x, y) is \{ \begin{matrix}an ellipse & if Δ < 0 \\ a parabola & if Δ = 0 \\ a hyperbola & if Δ > 0 \end{matrix}  (3.90)

When n = 3, the quadratic equation represents an ellipsoid. In general when n > 3, the equation N (x, m, Σ) = c_{0} represents a hyper ellipsoid in the ndimensional space. The center and spatial distribution of this ellipsoid are determined by m and Σ, respectively.

Specially if x = [x_{1}, ⋯, x_{n}]^{T} is decorrelated, i.e., σ_{ij} = 0 for all i ≠ j, Σ becomes a diagonal matrix

Σ = diag[σ_{1}^{2}, ⋯, σ_{n}^{2}] = [ \begin{matrix}σ_{1}^{2} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & σ_{n}^{2} \end{matrix} ]  (3.91)

and equation N (x, m, Σ) = c_{0} can be written as

(x − m)^{T} Σ^{−1}(x − m) = ∑_{i=1}^{n} \frac{(x_{i} − μ_{i})^{2}}{σ_{i}^{2}} = c_{1}  (3.92)

which represents a standard ellipsoid with all its axes parallel to those of the coordinate system.

Properties of Gaussian distribution • Sum of normal distributions:

If p(x_{i}) = N (m_{i}, Σ_{i}) (i = 1, 2), then p(x_{1} +x_{2}) = N (m_{1} +m_{2}, Σ_{1} +Σ_{2})

• Affine transformation of normal distributions: If p(x) = N (m_{x}, Σ_{x}), then p(y) = p(Ax + b) = N (m_{y}, Σ_{y}), where

m_{y} = Am_{x} + b, Σ_{y} = AΣ_{x}A^{T}  (3.93)

Proof:

m_{y} = E(Ax + b) = A E(x) + b = Am_{x} + b  (3.94)
Σ_{y} = E[(y − m_{y})(y − m_{y})^{T}]
= E[[(Ax − b) − (Am_{x} − b)][(Ax − b) − (Am_{x} + b)]^{T}]
= E[(Ax − Am_{x})(Ax − Am_{x})^{T}] = AE[(x − m_{x})(x − m_{x})^{T}]A^{T}
= AΣ_{x}A^{T}  (3.95)

Some special caess:

– If A = I, then p(x + b) = N (m_{x} + b, Σ_{x})

– If b = 0, then p(Ax) = N (m_{x}, AΣ_{x}A^{T})

– If p(x) = N ([μ_{1}, ⋯, μ_{n}]^{T} , diag[σ_{1}^{2}, ⋯, σ_{n}^{2}]), then p(∑_{i=1}^{n} x_{i}) = N (∑_{i=1}^{n} μ_{i}, ∑_{i=1}^{n} σ_{i}^{2})

• Marginal distributions:

p(x_{1}, x_{2}) = p(x) = p ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]) = N ([ \begin{matrix}μ_{1} \\ μ_{2} \end{matrix} ] , [ \begin{matrix}σ_{1}^{2} & σ_{12}^{2} \\ σ_{21}^{2} & σ_{2}^{2} \end{matrix} ]) (3.96)

Applying A_{1} = [1, 0] and A_{2} = [0, 1] to x = [x_{1}, x_{2}]^{T}, we get the two marginal distributions for A_{1}x = x_{1} and A_{2}x = x_{2}, respectively:

p(x_{1}) = N (μ_{1}, σ_{1}^{2}), p(x_{1}) = N (μ_{2}, σ_{2}^{2})  (3.97)

• Product of Gaussians: Consider two Gaussians:

p_{i}(x) = N (x, m_{i}, Σ_{i}) = \frac{1}{(2π)^{n/2}|Σ_{i}|^{1/2}} exp [−\frac{1}{2}(x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i})]
= \frac{1}{(2π)^{n/2}|Σ_{i}|^{1/2}} exp [−\frac{1}{2}(x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i} + m_{i}Σ_{i}^{−1}m_{i})]
= exp [−\frac{1}{2}[n log(2π) + log |Σ_{i}| + m_{i}Σ_{i}^{−1}m_{i}]] exp [−\frac{1}{2}(x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i})]
= exp [−\frac{1}{2}[x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i} + c_{i}]] (i = 1, 2)  (3.98)

where

c_{i} = n log(2π) + log |Σ_{i}| + m_{i}Σ_{i}^{−1}m_{i}  (3.99)

The product of two Gaussians is

p_{1}(x)p_{2}(x) = N (x, m_{1}, Σ_{2})N (x, m_{2}, Σ_{2})
= exp [−\frac{1}{2}[x^{T} Σ_{1}^{−1}x − 2x^{T} Σ_{1}^{−1}m_{1} + c_{1} + x^{T} Σ_{2}^{−1}x − 2x^{T} Σ_{2}^{−1}m_{2} + c_{2}]]
= exp [−\frac{1}{2}[x^{T} (Σ_{1}^{−1} + Σ_{2}^{−1})x − 2x^{T} (Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) + c_{1} + c_{2}]]
= exp [−\frac{1}{2}[x^{T} Σ^{−1}x − 2x^{T} Σ^{−1}m + c + (c_{1} + c_{2} − c)]]
= N (x, m, Σ) exp(c_{1} + c_{2} − c) ∝ N (x, m, Σ)  (3.100)

Here we have defined

c = n log(2π) + log |Σ| + m^{T} Σ^{−1}m  (3.101)
Σ = (Σ_{1}^{−1} + Σ_{2}^{−1})^{−1}  (3.102)

and

Σ^{−1}m = Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}  (3.103)

i.e.,

m = Σ(Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) = (Σ_{1}^{−1} + Σ_{2}^{−1})^{−1}(Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) (3.104)

We see that the product of Gaussians is also a scaled Gaussian.

• Marginal and conditional distributions of multivariate normal dis-

tribution

A random vector x = [x_{!}^{T} , x_{2}^{T}]^{T} with a normal distribution N (x, m, Σ) can be partitioned into two subvectors x_{1} and x_{2}:

p ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]) = p(x) = N (x, m, Σ) = N ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , [ \begin{matrix}m_{1} \\ m_{2} \end{matrix} ] , [ \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ])

(3.105)

where Σ_{ij} = Σ_{ji}^{T}, Σ_{ii} = Σ_{ii}^{T}, (i, j = 1, 2, i ≠ j). Given this normal joint distribution of x_{1} and x_{2}, we can find the normal conditional distribution of x_{i} given x_{j}:

p(x_{i}/x_{j}) = N (x_{i}, m_{i/j}, Σ_{i/j})  (3.106)

in terms of the mean vector and the covariance matrix:

\{ \begin{matrix}m_{i/j} & = & m_{i} + Σ_{ij}Σ_{jj}^{−1}(x_{j} − m_{j}) \\ Σ_{i/j} & = & Σ_{ii} − Σ_{ij}Σ_{jj}^{−1}Σ_{ji} \end{matrix}  (3.107)

Proof: The joint distribution of x can be written as:

p(x) = p(x_{1}, x_{2}) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)]
= \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}Q(x_{1}, x_{2})]  (3.108)

Here we have defined:

Q(x_{1}, x_{2}) = (x − m)^{T} Σ^{−1}(x − m)
= [(x_{1} − m_{1})^{T} , (x − m_{2})^{T}] [ \begin{matrix}Σ^{11} & Σ^{12} \\ Σ^{21} & Σ^{22} \end{matrix} ] [ \begin{matrix}x_{1} − m_{1} \\ x_{2} − m_{2} \end{matrix} ]
= (x_{1} − m_{1})^{T} Σ^{11}(x_{1} − m_{1}) + 2(x_{1} − m_{1})^{T} Σ^{12}(x_{2} − m_{2}) + (x_{2} − m_{2})^{T} Σ^{22}(x_{2} − m_{2}) (3.109)

where we have assumed

Σ^{−1} = [ \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ]^{−1} = [ \begin{matrix}Σ^{11} & Σ^{12} \\ Σ^{21} & Σ^{22} \end{matrix} ]  (3.110)

According to the Woodbury matrix identify (Section ?? in Appendix A), we have

Σ^{11} = (Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ_{21})^{−1} = Σ_{11}^{−1} + Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}
Σ^{22} = (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1} = Σ_{22}^{−1} + Σ_{22}^{−1}Σ_{21}(Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ_{21})^{−1}Σ_{12}Σ_{22}^{−1}
Σ^{12} = −Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1} = (Σ^{12})  (3.111)

Substituting the second expression for Σ^{11}, the first expression for Σ^{22}, and Σ^{12} into Q(x_{1}, x_{2}) we get:

Q(x_{1}, x_{2}) = (x_{1} − m_{1})^{T} [Σ_{11}^{−1} + Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}](x_{1} − m_{1})

−2(x_{1} − m_{1})^{T} [Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}](x_{2} − m_{2})

+(x_{2} − m_{2})^{T} [(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}](x_{2} − m_{2})

= (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})

+(x_{1} − m_{1})^{T} Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})

−2(x_{1} − m_{1})^{T} Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}(x_{2} − m_{2})

+(x_{2} − m_{2})^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}(x_{2} − m_{2})

= (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})

+[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})] (3.112)

The last equal sign is due to the following equations for any vectors u and v and a symmetric matrix A = A^{T}:

u^{T} Au − 2u^{T} Av + v^{T} Av = u^{T} Au − u^{T} Av − u^{T} Av + v^{T} Av
= u^{T} A(u − v) − (u − v)^{T} Av = u^{T} A(u − v) − v^{T} A(u − v)
= (u − v)^{T} A(u − v) = (v − u)^{T} A(v − u)  (3.113)

We further define

\{ \begin{matrix}b = m_{2} + Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1}) \\ A = Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12} \end{matrix}  (3.114)

and

Q_{1}(x_{1})  = (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})
Q_{2}(x_{1}, x_{2}) = [(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]
= (x_{2} − b)^{T} A^{−1}(x_{2} − b)

(3.115)

then the expression above for Q(x_{1}, x_{2}) can be written as:

Q(x_{1}, x_{2}) = Q_{1}(x_{1}) + Q_{2}(x_{1}, x_{2})  (3.116)

and the joint distribution can be written as:

p(x) = p(x_{1}, x_{2}) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}Q(x_{1}, x_{2})]
= \frac{1}{(2π)^{n/2}|Σ_{11}|^{1/2}|Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12}|^{1/2}} exp [−\frac{1}{2}Q_{1}(x_{1})] exp [−\frac{1}{2}Q_{2}(x_{1}, x_{2})]
= \frac{1}{(2π)^{p/2}|Σ_{11}|^{1/2}} exp [−\frac{1}{2}(x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})] \frac{1}{(2π)^{q/2}|A|^{1/2}} exp [−\frac{1}{2}(x_{2} − b)^{T} A^{−1}(x_{2} − b)]
= N (x_{1}, m_{1}, Σ_{11}) N (x_{2}, b, A)  (3.117)

where we have used theorem 2 in Section 7 in Appendix A:

|Σ| = |Σ_{11}| |Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12}|  (3.118)

The marginal distribution of x_{1} is

p_{1}(x_{1}) = ∫ p(x_{1}, x_{2})dx_{2} = N (x_{1}, m_{1}, Σ_{11}) ∫ N (x_{2}, b, A)dx_{2}
= N (x_{1}, m_{1}, Σ_{11}) = \frac{1}{(2π)^{p/2}|Σ_{11}|^{1/2}} exp [−\frac{1}{2}(x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})] (3.119)

and the conditional distribution of x_{2} given x_{1} is

p_{2|1}(x_{2}|x_{1}) = \frac{p(x_{1}, x_{2})}{p_{1}(x_{1})} = N (x_{2}, b, A) = \frac{1}{(2π)^{q/2}|A|^{1/2}} exp [−\frac{1}{2}(x_{2} − b)^{T} A^{−1}(x_{2} − b)]

(3.120)

with

\{ \begin{matrix}b & = & m_{2/1} = m_{2} + Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1}) \\ A & = & Σ_{2/1} = Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12} \end{matrix}  (3.121)

Drawing samples from Gaussian distributions We first consider drawing samples from a random variable x with a normal

distribution p(x) = N (0, 1), i.e., E(x) = 0 and V ar(x) = E(x^{2}) = 1. As shwon in the figure, a random variable uniformly distributed in the range (0, 1) (vertical) can be mapped to another random variable x in the range (−∞, ∞) (horizontal) with a Gaussian distribution p(x) = N (0, 1), by the inverse function of its accumulative F (x) = ∫_{−∞}^{x} p(u) du. This variable x can be further converted into another variable x′ = μ + σx, whose distribution is a generic Gaussian p(x′) = N (μ, σ^{2}), with the mean and variance

E(x′) = E(μ + σx) = μ + σE(x) = μ
V ar(x′) = [(x′ − μ)^{2}] = E(σ^{2}x) = E(σ^{2}x^{2}) = σ^{2}E(x^{2}) = σ^{2} (3.122)

Using this method, we can generate a set of n normally distributed and independent random variables x_{1}, ⋯, x_{n}, each with E(x_{i}) = 0, V ar(x_{i}) = 1, and Cov(x_{i}, x_{j}) = δ_{ij}, and, based on these variables, we can further construct a random vector x = [x_{1}, ⋯, x_{n}]^{T} with a Gaussian joint distributed p(x) = N (0, I).

Given such an x with m_{x} = 0 and Σ_{x} = I, we can generate another random vector with a Gaussian distribution N(m, Σ). To do so, consider the affine transformation y = Wx + m, which also has a Gaussian distribution:

m_{y} = E(y) = Wm_{x} + m = m
Σ_{y} = WΣ_{x}W^{T} = WW^{T}  (3.123)

While m_{y} = m as desired, we still need to make sure that Σ_{y} = WW^{T} = Σ also as desired. There exist infinite matricies all satisfying the equation above. Typically, such a W can be found in either of the following two methods:

• The Cholesky decomposition: which can find an upper trangular matrix L that satisfies Σ = LL^{T} (if Σ is nonsingular).

• The eigenvalue decomposition Σ = VΛV^{T} (even if Σ is singular), where V and Λ are respectively the eigenvector and eigenvalue matrices of Σ. As Σ^{T} = Σ is symmetric and positive definite, its eigenvalues take positive real values and its eigenvectors are orthogonal to each other, i.e., V^{T} = V^{−1} is orthogonal. If we let W = VΛ^{1/2}, then we have

WW^{T} = (VΛ^{1/2}) (VΛ^{1/2})^{T} = VΛV^{T} = Σ  (3.124)

We note that although LL^{T} = WW^{T} = Σ, L found by the Cholesky decomposition method is upper trangular but W is not, L ≠ W. Also, we note that the transform based on eigenvalue decomposition is optimal in certain way, as to be discussed later.

Having found either L or W, we get y = Lx + m or y = Wx + m with p(y) = N (m, Σ) as desired.

3.1.4 Function of Random Variables

We first find the mean and variance of y = cx where x is a random variable and c is the constant:

μ_{y} = E(y) = E(cx) = c E(x) = c μ_{x}
σ_{y} = E(y^{2}) − μ_{y}^{2} = E[(cx)^{2}] − (cμ_{x})^{2} = c^{2} E(x^{2}) − c^{2}μ_{x}^{2} = c^{2}(E(x^{2}) − μ_{x}^{2}) = σ_{x}^{2} (3.125)

We next consider the mean and variance of y = x_{1} + ⋯ + x_{n} as the sum of n random variables:

μ_{y} = E(y) = E (∑_{i=1}^{n} x_{i}) = ∑_{i=1}^{n} E(x_{i}) = ∑_{i=1}^{n} μ_{x_{i}}  (3.126)

We also have

E(y^{2}) = E (∑_{i=1}^{n} x_{i})^{2} = E (∑_{i=1}^{n} ∑_{j=1}^{n} x_{i}x_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} E(x_{i}x_{j})  (3.127)

and

(E(y))^{2} = (∑_{i=1}^{n} E(x_{i}))^{2} = ∑_{i=1}^{n} ∑_{j=1}^{n} E(x_{i})E(x_{j})  (3.128)

then we get

V ar(y) = E(y^{2})−(E(y))^{2} = ∑_{i=1}^{n} ∑_{j=1}^{n} E(x_{i}x_{j})−∑_{i=1}^{n} ∑_{j=1}^{n} E(x_{i})E(x_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} Cov(x_{i}, x_{j})

(3.129)

Only if all x_{i} are uncorrelated (or more specially independent), i.e., Cov(x_{i}, x_{j}) = 0 for i ≠ j, do we have

σ_{y}^{2} = V ar(y) = ∑_{i=1}^{n} ∑_{j=1}^{n} Cov(x_{i}, x_{j}) = ∑_{i=1}^{n} Cov(x_{i}, x_{i}) = ∑_{i=1}^{n} V ar(x_{i}) = ∑_{i=1}^{n} σ_{x_{i}}^{2}

(3.130)

We further consider in general y = f(x) as a nonlinear function of a random variable x is also a random variable, which can be approximated based on its Taylor expansion near the mean μ_{x} of x:

y = f(x) = f(μ_{x})+f′(μ_{x})(x−μ_{x})+ \frac{f′′(μ_{x})}{2}(x−μ_{x})^{2}+⋯ ≈ f(μ_{x})+f′(μ_{x})(x−μ_{x})

(3.131)

The approximation is due to the assumption that |x − μ_{x}| is small. Taking the mean on both side we get

μ_{y} = E(y) ≈ E[f(μ_{x}) + f′(μ_{x})(x − μ_{x})] = f(μ_{x})  (3.132)

Taking the variance on both sides we get

V ar(y) ≈ V ar[f(μ_{x})+f′(μ_{x})(x−μ_{x})] = V ar[f′(μ_{x})(x−μ_{x})] = f^{′2}(μ_{x})V ar(x) = f^{′2}(μ_{x})σ_{x}^{2}

(3.133)

Finally to find the mean and variance of y = f(x_{1}, ⋯, x_{n}) = f(x) as a nonlinear function of n random variables represented as an n-D vector x = [x_{1}, ⋯, x_{n}]^{2}, consider its Taylor expansion in the neighborhood of the mean vector m_{x} = [μ_{x_{1}}, ⋯, μ_{x_{n}}]^{T}:

f(x_{1}, ⋯, x_{n}) = f(μ_{x_{1}}, ⋯, μ_{x_{n}}) + ∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}}) + ∑_{i=1}^{n} ∑_{j=1}^{n} \frac{∂f^{2}}{∂x_{i}∂x_{j}} (x_{i} − μ_{x_{i}})(x_{j} − μ_{x_{j}}) + ⋯

≈ f(μ_{x_{1}}, ⋯, μ_{x_{n}}) + ∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}})  (3.134)

or in matrix form:

f(x) = f(m_{x})+g(m_{x})^{T} (x−m_{x})+\frac{1}{2}(x−m_{x})^{T} H(m_{x})(x−m_{x})+⋯ ≈ f(m_{x})+g(m_{x})^{T} (x−m_{x})

(3.135)

where

g(m_{x}) = [ ⋮_{\frac{∂fx)}{∂x_{n}}}^{\frac{∂fx)}{∂x_{1}}} ]_{m_{x}} , H(m_{x}) = [ \begin{matrix}\frac{∂^{2}f(x)}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x)}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{n}^{2}} \end{matrix} ]_{m_{x}} (3.136)

are the gradient vector and Hessian matrix of f(x) evaluated at m_{x}. Taking the mean of the expression above

μ_{y} = Ef(x) ≈ E [f(μ_{x_{1}}, ⋯, μ_{x_{n}}) + ∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}})]

= f(μ_{x_{1}}, ⋯, μ_{x_{n}}) + ∑_{i=1}^{n} \frac{∂f}{∂x_{i}} E(x_{i} − μ_{x_{i}}) = f(μ_{x_{1}}, ⋯, μ_{x_{n}}) (3.137)

Taking the variance we get

σ_{y}^{2} = V ar [f((x_{1}, ⋯, x_{n})] ≈ V ar [f(μ_{x_{1}}, ⋯, μ_{x_{n}}) + ∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}})]

= V ar [∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}})] = ∑_{i=1}^{n} (\frac{∂f}{∂x_{i}} )^{2} σ_{x_{i}}^{2}  (3.138)

or

σ_{y} = \sqrt{∑_{i=1}^{n} (\frac{∂f}{∂x_{i}} )^{2} σ_{x_{i}}^{2}}  (3.139)

3.2 Entropy and Information

3.2.1 Entropy

Entropy is a measurement of the uncertainty of the outcome of a random event, which also measures the amount of information obtained in terms of the reduction of uncertainty (due to certain communication or xperiment). If the uncertainty is reduced to zero by a communication, then the amount of information obtained is exactly the same as the uncertainty before the communication.

A random event of n outcomes can be represented by a discrete random variable x with N possible values {x_{1}, ⋯, x_{n}} with the corresponding probabilities P_{i} = P (x_{i}) satisfying ∑_{i=1}^{n} P_{i} = 1. For example, n = 26 letters in the English alphabet, with different probabilities to be used (e.g., P (x =′ a′) > P (x =′ z′)).

To quantitatively measure the uncertainty of random variable x denoted by H(x), we define the uncertainty or information content (surprise) of each outcome x_{i} as

H(x_{i}) = log \frac{1}{P_{i}} = − log P_{i}, (i = 1, ⋯, n)  (3.140)

so that the desired properties listed below are satisfied:

• Uncertainty is inversely related to the probability, i.e., a more probable outcome is less uncertain than a less probable outcome. If P_{i} > P_{j}, then

H(x_{i}) = − log P_{i} < H(x_{j}) = log P_{j}  (3.141)

• Specially, an certain outcome x with P (x_{i}) = 1 has zero uncertainty, and an impossible outcome x with P (x_{j}) = 0 has infinite uncertainty:

H(x_{i}) = − log P_{i} = − log 1 = 0, H(x_{j}) = − log P_{j} = − log 0 = ∞

(3.142)

• The total uncertainty of x is the expectation of the uncertainties of the n outcomes:

H(x) = E[H(x_{i})] = E [− log P_{i}] = − ∑_{i=1}^{n} P_{i} log P_{i}  (3.143)

The total uncertainty H(x) defined above, called the entropy, measures the total uncertainty of the random event represented by x, which is also the total amount of information contained in x. Once the outcome of the random event is known, the information contained in x is gained, and the uncertainty is reduced to zero.

The n outcomes of x can be binary encoded so that each outcome x_{i} is represented by a sequence of 0’s and 1’s. To minimizes the total number of bits needed to encode x, we encode x_{i} with P_{i} by H(x_{i}) = − log_{2} P_{i} bits, so that a more probable outcome x_{i} with greater P_{i} (less uncertain and containing less information) is encoded by fewer bits, and a less probable outcome x_{j} with smaller P_{j} (more uncertain and containing more information) is encoded by more bits. Then the expected number of bits needed to encode x is minimized.

We can show that a uniform distribution of n = 2^{m} equalily likely outcomes with p_{i} = 1/n (i = 1, ⋯, n) has the maximum entropy. This is a constrained maximization problem which can be solved by Lagrange multiplier method:

\frac{∂}{∂P_{j}} [− ∑_{i=1}^{n} P_{i}log P_{i} + λ (1 − ∑_{i=0}^{n} P_{i})] = 0

= −\frac{∂}{∂P_{j}} P_{j} log P_{j} − λ = −(log P_{j} + 1) − λ = 0  (3.144)

Solving this we get

P_{j} = 2^{−λ−1} (j = 1, ⋯, N)  (3.145)

which must satisfy

∑_{j=1}^{n} P_{j} = ∑_{j=1}^{n} 2^{−λ−1} = 2^{−λ−1}n = 1  (3.146)

Solving for λ we get

λ = log n − 1  (3.147)

Substituting back into P_{j} we get

P_{j} = 2^{−λ−1} = 2^{− log n} = \frac{1}{n}, (j = 1, ⋯, n)  (3.148)

The corresponding maximum entropy can be obtained as

H_{max}(E_{1}, ⋯, E_{n}) = − ∑_{i=1}^{n} \frac{1}{n}log_{2} \frac{1}{n} = log_{2}n = m  (3.149)

In this case, each outcome x_{i} is encoded by

log_{2} \frac{1}{P_{i}} = log_{2} (\frac{1}{1/2^{n}} ) = log_{2} 2^{n} = n  (3.150)

bits, and the total number of bits needed to encode this random event is

− ∑_{i=1}^{n} P_{i} log_{2} P = − ∑_{i=1}^{n} \frac{1}{n} log_{2} \frac{1}{2^{m}} = ∑_{i=1}^{n} \frac{1}{n} log_{2} 2^{m} = m  (3.151)

If n = 2^{m} = 2^{3} = 8, P_{i} = 1/2^{3} = 1/8, n = 3 bits are needed to encode the n = 8 possible outcomes.

For example, in the special case of n = 2 outcomes:

• If P_{1} = 1 and P_{2} = 0, we have the minimum uncertainty zero (x log_{2} x|_{x=0} = 0), i.e., 0 bit is needed to encode a sure event:

H = −1 log_{2} 1 − 0 log_{2} 0 = 0  (3.152)

• If P_{1} = 0.1, P_{2} = 0.9, the uncertainty is:

H = −(0.1 log_{2} 0.1 + 0.9 log_{2} 0.9) = 0.47  (3.153)

• If P_{1} = 0.2, P_{2} = 0.8, the uncertainty is:

H = −(0.2 log_{2} 0.2 + 0.8 log_{2} 0.8) = 0.72  (3.154)

• If P_{1} = 0.3, P_{2} = 0.7, the uncertainty is:

H = −(0.3 log_{2} 0.3 + 0.7 log_{2} 0.7) = 0.88  (3.155)

• If P_{1} = 0.4, P_{2} = 0.6, the uncertainty is:

H = −(0.4 log_{2} 0.4 + 0.6 log_{2} 0.6) = 0.97  (3.156)

• If P_{1} = P_{2} = 0.5 = 1/2, we have the maximum uncertainty H = 1, i.e., 1 bit (0 or 1) is needed to encode the event:

H = −\frac{1}{2} log_{2} \frac{1}{2} − \frac{1}{2} log_{2} \frac{1}{2} = \frac{1}{2} + \frac{1}{2} = 1  (3.157)

The discussion above for a discrete random variable x can be entended to a continuous random variable x with pdf p(x), of which the differential entropy is defined as

H(p) = − ∫ p(x) log p(x) dx  (3.158)

The perplexity of a discrete probability distribution P is simply defined as 2^{H(p)}.

We consider the pdf p(x) that maximizes the entropy under each of the following two conditions:

• The p(x) is non-zero only inside a closed region x ∈ [a, b].

To find the pdf p(x) that maximizes H(p) under the condition ∫_{a}^{b} p(x) dx = 1, we construct its Lagrangian function

L(p) = H(p) + λ (1 − ∫_{a}^{b} p(x) dx) = ∫_{a}^{b} [−p(x) log p(x) − λp(x)] dx + λ

(3.159)

and set its derivative with respect to p(x) (with a specific value of x) to zero:

\frac{d L(p)}{d p(x)} = −(log p(x) + 1) − λ = 0  (3.160)

Solving for p(x) we get

p(x) = e^{−λ−1}  (3.161)

which must satisfy

∫_{−∞}^{∞} p(x) dx = ∫_{a}^{b} e^{−λ−1} dx = e^{−λ−1}(b − a) = 1  (3.162)

i.e.,

p(x) = e^{−λ−1} = \frac{1}{b − a}  (3.163)

This is a uniform distribution over [a, b]. The corresponding maximum entropy is

H(p) = − ∫_{a}^{b} \frac{1}{b − a} log (\frac{1}{b − a}) = log(b − a)  (3.164)

• The random variable x ∈ (−∞, ∞) with a known vairance

σ^{2} = ∫_{−∞}^{∞}(x − μ)^{2}p(x) dx  (3.165)

To find the pdf p(x) with σ^{2} that maximizes H(p), we consider the following Lagrangian function

L(p) = − ∫_{−∞}^{∞} p(x) log p(x) dx+λ_{1} (1 − ∫_{−∞}^{∞} p(x) dx)+λ_{2} (σ^{2} − ∫_{−∞}^{∞}(x − μ)^{2} p(x) dx)

(3.166)

and set its derivative with respect to p(x) (evaluated at certain x) to zero:

\frac{d L(p)}{d p(x)} = −(log p(x) + 1) − λ_{1} − λ_{2}(x − μ)^{2} = 0  (3.167)

Solving for p(x) we get

p(x) = e^{−(λ_{1}+λ_{2}(x−μ)^{2}+1)} = e^{−(λ_{1}+1)}e^{−λ_{2}(x−μ)^{2}}  (3.168)

which must satisfy the two constraints:

1 = ∫_{−∞}^{∞} p(x) dx = e^{−λ_{1}−1} ∫_{−∞}^{∞} e^{−λ_{2}(x−μ)^{2}}dx = e^{−(λ_{1}+1)}\sqrt{\frac{π}{λ_{2}}} (3.169)

and

σ^{2} = ∫_{−∞}^{∞}(x − μ)^{2}p(x) dx = e^{−λ_{1}−1} ∫_{−∞}^{∞} e^{−λ_{2}(x−μ)^{2}}(x − μ)^{2} dx
= e^{−λ_{1}−1} \frac{1}{2}\sqrt{\frac{π}{λ_{2}^{3}}} = e^{−λ_{1}−1} \frac{1}{2λ_{2}} \sqrt{\frac{π}{λ_{2}}}  (3.170)

Comparing the two equations above, we get

λ_{2} = \frac{1}{2σ^{2}}  (3.171)

Substituting this into Eq. (3.169), we get

e^{−(λ_{1}+1)}\sqrt{2πσ^{2}} = 1,  i.e.  e^{−(λ_{1}+1)} = \frac{1}{\sqrt{2πσ^{2}}}  (3.172)

Substituting this together with λ_{2} = 1/2σ^{2} into p(x) we get

p(x) = e^{−(λ_{1}+1)}e^{−λ_{2}(x−μ)^{2}} = \frac{1}{\sqrt{2πσ^{2}}} e^{− \frac{(x−μ)^{2}}{2σ^{2}}} = N (x, μ, σ^{2}) (3.173)

This is the normal or Gaussian distribution and the corresponding maximum entropy is

H(g) = − ∫_{−∞}^{∞} g(x) log g(x) dx = − ∫_{−∞}^{∞} g(x) log (\frac{1}{\sqrt{2πσ^{2}}} e^{−(x−μ)^{2}/2σ^{2}}) dx
= − ∫_{−∞}^{∞} g(x) (log \frac{1}{\sqrt{2πσ^{2}}} − \frac{(x − μ)^{2}}{2σ^{2}} ) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∫_{−∞}^{∞}(x − μ)^{2}g(x) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} σ^{2} = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} log(e)
= \frac{1}{2} log(2πeσ^{2}) = \frac{1}{2} log(2πe) + log σ  (3.174)

The uncertainty of a distribution p(x) can be measured by both its variance σ^{2} = ∫ (x − μ)^{2}p(x) dx and its entropy H = − ∫ p(x) log p(x) dx. However, they are also different in that σ^{2} represents the spread of p(x) while H captures the shape of p(x). In multimodal distribution (with two or more modes), H is a better measurement as it increases due to the multimodes while σ^{2} is not sensitive to them.

In data analysis and modeling, if no additional knowledge is available regarding the observed data other than the variance (or covariance for multivariate data), the normal distribution with maximum entropy should be used as leastinformative default to model the data, as it will impose the least amount of unsupported constraint and thereby causing minimum bias. This is the principle of maximum entropy.

• Joint Entropy The joint entropy of two discrete random variables x and y is defined as:

H(x, y) = −E[log P (x_{i}, y_{j})] = − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i}, y_{j}) (3.175)

More generally, the joint entropy of n discrete random variables in x = [x_{1}, ⋯, x_{n}]^{T} is defined as:

H(x) = H(x_{1}, ⋯, x_{n}) = −E[log P (x_{1}, ⋯, x_{n})]
= − ∑_{x_{1}} ⋯ ∑_{x_{n}} P (x_{1}, ⋯, x_{n}) log P (x_{1}, ⋯, x_{n})
= − ∑_{x_{1}} ⋯ ∑_{x_{n}} P (x) log P (x)  (3.176)

The differential entropy is

H(x) = H(x_{1}, ⋯, x_{n}) = − ∫ ⋯ ∫ p(x_{1}, ⋯, x_{n}) log P (x_{1}, ⋯, x_{n})dx_{1} ⋯ dx_{n}
= − ∫ p(x) log P (x)dx  (3.177)

• Entropy of function of random variables

Given the entropy H(x) of x, we can further find the entropy of a function y = f(x) of x. We first recall that the pdf of y can be expressed in terms of that of x in Eq. (3.74): p_{Y} (y) = | det J_{f^{−1}}|p_{X}(x). We can then find H(y) directly as:

H(y) = − ∫ p_{y}(y) log p_{y}(y)dy
= − ∫ | det J_{f^{−1}}| p_{X}(x) log (| det J_{f^{−1}}| p_{X}(x)) | det J_{f}| dx
= − ∫ p_{X}(x) [log | det J_{f^{−1}}| + log p_{X}(x)] dx
= − ∫ p_{X}(x) log p_{X}(x) dx − ∫ p_{X}(x) log | det J_{f^{−1}}| dx
= H(x) + E [log | det J_{f}|]  (3.178)

Here we have used integration by substitution and also the inverse function theorem stating that given the Jacobian J_{f} of a function y = f(x), the Jacobian of the inverse function x = f^{−1}(y) is J_{f^{−1}} = J_{f}^{−1}. Also as in general det(A^{−1}) = (det(A))^{−1}, we have | det J_{f^{−1}}| | det J_{f}| = 1.

Specially for a linear function y = f(x) = Ax, we have J_{f} = df(x)/dx = A and J_{f^{−1}} = A^{−1}, then we have

H(y) = H(x) + log | det A|  (3.179)

• Conditional Entropy

The conditional entropy H(y|x) measures the uncertainty of a random variable y given the value of another random variable x, or the amount of information gained once the outcome of y is known, given the outcome of x.

Let H(y|x = x_{i}) be the entropy of y conditioned on x = x_{i}:

H(y|x = x_{i}) = − ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i})  (3.180)

Then the conditional entropy H(y|x) is defined as the average of H(y|x = x_{i}) over all outcomes of x:

H(y|x) = E_{x}[H(y|x = x_{i})] = ∑_{i} P (x_{i})H(y|x = x_{i})
= − ∑_{i} P (x_{i}) ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i}) = − ∑_{i} ∑_{j} P (x_{i})P (y_{j}|x_{i}) log P (y_{j}|x_{i})
= − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j}|x_{i})  (3.181)

The last equality is due to P (x, y) = P (y|x)P (x).

Theorem:

H(y|x) = − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j}|x_{i}) = ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i})}{P (x_{i}, y_{j})}
= − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i}, y_{j}) + ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i})
= H(x, y) + ∑_{i} P (x_{i}) log P (x_{i}) = H(x, y) − H(x)  (3.182)

Similarly, we also have H(x|y) = H(x, y) − H(y). We therefore have

H(x, y) = H(y|x) + H(x) = H(x|y) + H(y)  (3.183)

Substituting this into the previous equation we get Bayes’s rule for conditional entropy:

H(y|x) = H(x|y) + H(y) − H(x)  (3.184)

3.2.2 Mutual Information and Kullback-Leibler Divergence

• Mutual Information

Mutual information I(x, y) measures the information shared by random variables x and y, i.e., how much knowing one of the two variables reduces uncertainty of the other.

I(x, y) = ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i}, y_{j})}{P (x_{i})P (y_{j})}
= ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i}, y_{j})}{P (x_{i})} − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j})

= ∑_{i} ∑_{j} P (y_{j}|x_{i})P (x_{i}) log P (y_{j}|x_{i}) − ∑_{j} log P (y_{j}) (∑_{i} P (x_{i}, y_{j}))
= ∑_{i} P (x_{i}) ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i}) − ∑_{j} P (y_{j}) log P (y_{j})
= −H(y|x) + H(y) = H(y) − H(y|x)
= H(x, y) − H(y|x) − H(x|y) = H(x) + H(y) − H(x, y)  (3.185)

The last two equalities are due to H(x, y) = H(x|y) + H(y) above.

If P (x, y) = P (x) P (y), then log P (x, y)/P (x)P (y) = log 1 = 0 and I(x, y) = 0, i.e., the mutual information of two independent random variables is zero.

These relationships can be summarized in the following diagram:

The concept of mutual information can be generalized from two variables to multiple variables:

I(x) = I(x_{1}, ⋯, x_{n}) = ∑_{i_{1}} ⋯ ∑_{i_{n}} p(x) log \frac{∏_{j=1}^{n} p(x_{j})}{p(x)}

= ∑_{i=1}^{n} H(x_{i}) − H(x_{1}, ⋯, x_{n}) = ∑_{i=1}^{n} H(x_{i}) − H(x) (3.186)

• Cross Entropy

When the true distribution P is unknown, the encoding of x can be based on another distribution Q as a model that approximates P. Then the average of the total number of bits needed is called the cross-entropy:

H(P, Q) = ∑_{i} P_{i} log_{2} \frac{1}{Q_{i}} = − ∑_{i} P_{i} log Q_{i} = −E_{P} [log Q]  (3.187)

which can be considered as the expectation of the logrithmic model probability log Q_{i} with respect to the unknown ground truth probability P_{i}.

Gibbs’ inequality: The cross entropy H(P, Q) is no smaller than the entropy H(P):

H(P) = − ∑_{i} P_{i} log P_{i} ≤ H(P, Q) = − ∑_{i} P_{i} log Q_{i}  (3.188)

The equality holds H(P, Q) = H(P) if and only if Q = P. For the model Q to be as close to the ground truth P as possible, we need to minimize the cross entropy H(P, Q).

Proof: Based on inequality log x ≤ x − 1, we have

∑_{i} P_{i} log (\frac{Q_{i}}{P_{i}} ) ≤ ∑_{i} P_{i} (\frac{Q_{i}}{P_{i}} − 1) = ∑_{i} Q_{i} − ∑_{i} P_{i} = 0 (3.189)

i.e.,

∑_{i} P_{i} log (\frac{Q_{i}}{P_{i}} ) = ∑_{i} P_{i} log (\frac{1}{P_{i}} ) + ∑_{i} P_{i} log Q_{i} = H(P) − H(P, Q) ≤ 0

(3.190)

We see that the encoding of x based on model distribution Q always requires more bits than that based on the true distribution P.

• Kullback-Leibler Divergence (Relative Entropy)

The Kullback-Leibler (KL) divergence or relative entropy is the difference between the cross entropy H(P, Q) and the entropy H(P):

D_{KL}(P ||Q) = H(P, Q) − H(P) = −E_{P} [log Q] − E_{P} [log P]
= − ∑_{i} P_{i} log Q_{i} − (− ∑_{i} P_{i} log P_{i})
= ∑_{i} P_{i}(log P_{i} − log Q_{i}) = ∑_{i} P_{i} log (\frac{P_{i}}{Q_{i}} ) ≥ 0(3.191)

The KL-divergence represents the number of extra bits needed to encode x based on Q instead of P, or a measure of the error of using Q to approximate P, in terms of the amount of information lost, due to the inaccuracy of the model. In order to obtain the best model Q that optimally approximates P, we need to minimize D_{KL}(P ||Q).

In general, D_{KL}(P ||Q) ≠ D_{KL}(Q||P), the KL-divergence is not a distance metric as it is not symmetric (hence the name divergence rather than distance). The KL-divergence can be symmetrized to become the Jensen–Shannon (JS) divergence:

JS(P ||Q) = \frac{1}{2} [D_{KL}(P ||M) + D_{KL}(Q||M)] , where M = \frac{P + Q}{2}

(3.192)

As JS divergence is symmetric, it can be used as a distance measure for the smilarity between two distributions P and Q.

The mutual information defined above can be expressed as the following KL-divergence:

I(x_{1}, ⋯, x_{n}) = ∑_{i_{1}} ⋯ ∑_{i_{n}} P (x_{1}, ⋯, x_{n}) log \frac{P (x_{1}, ⋯, x_{n})}{∏_{j=1}^{n} P (x_{j})} = D_{KL}(P (x)|| ∏_{j=1}^{n} P (x_{j}))

(3.193)

i.e., the mutual information is the error of using P (x) P (y) to model the joint probability P (x, y). When x and y are independent of each other, i.e.,

P (x, y) = P (x) P (y), P (x|y) = P (x), P (y|x) = P (y) (3.194)

their mutual information is equal to zero.

All discussions above for discrete variables can be generalized tocontinuous variables with summations replaced by integrals.

In particular, consider the cross-entropy of a Gaussian g(x) and an arbitrary pdf p(x) both with the same variance σ^{2}:

H(p, g) = − ∫ p(x) log g(x) dx = − ∫ p(x) (log \frac{1}{\sqrt{2πσ^{2}}} − \frac{(x − μ)^{2}}{2σ^{2}} ) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∫ (x − μ)^{2}p(x) dx = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} σ^{2}
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} log e = \frac{1}{2} log(2πeσ^{2}) = H(g) (3.195)

We see that the cross-entropy H(p, g) of g(x) and p(x) is the same as the entropy of g(x). But as we also know H(p, g) ≥ H(p), we have H(g) = H(p, g) ≥ H(p). We therefore conclude that among all pdfs p(x) with the same variance σ^{2}, the Gaussian N(x, μ, σ^{2}) has the maximum entropy.

3.2.3 Entropy Maximization

There may be the need to find an unknown distribution p(x) of a random varible x based on the observed data as the averages of set observed experiment results f_{i}(x):

E[f_{i}(x)] = ∫_{−∞}^{∞} p(x)f_{i}(x) dx = c_{i} (i = 1, ⋯, m)  (3.196)

Among the infinite number of density functions all satisfying these equations as well as ∫_{−∞}^{∞} p(x) dx = 1, the one with the maximum entropy H(p) is of the most interest. In the following we show that such a distribution has to take the following form:

p(x) = exp (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) = A exp (∑_{i=1}^{m} λ_{i}f_{i}(x))  (3.197)

where λ_{0}, ⋯, λ_{m} and A = e^{λ_{0}} are constants with values for p(x) tosatisfy Eq. (3.196).

To find this distribution p(x) with maximal H(p) and satisfying Eq. (3.196) as constraints, as well as ∫_{−∞}^{∞} p(x) dx = 1 required of all probability densityy distributions, we need to solve a constrainted optimization problem with the Lagrangian:

L(p) = ∫_{−∞}^{∞} p(x) log p(x) dx+λ_{0} (∫_{−∞}^{∞} p(x)dx − 1)+∑_{i=1}^{m} λ_{i} (∫_{−∞}^{∞} p(x)f_{i}(x) dx − 1)

(3.198)

We then take its derivative with respect to p(x) corresponding to a specific value of x (i.e., x is treated as constant and the integral is ignored), and set it to zero:

\frac{d}{dp} L(p) = \frac{d}{dp} (p log p) + λ_{0} \frac{d}{dp} (p − 1) + ∑_{i=1}^{m} λ_{i} \frac{d}{dp} (pf_{i}(x) − 1)

= log p(x) + 1 − λ_{0} − ∑_{i=1}^{m} λ_{i}f_{i}(x) = 0  (3.199)

Solving this equation we get the optimal distribution:

p(x) = exp (λ_{0} − 1 + ∑_{i=1}^{m} λ_{i}f_{i}(x))  (3.200)

which is indeed in the form of Eq. (3.197).

Alternatively, the same result can also be obtained by comparing the entropies of an arbitrary density function q(x), and another distribution p(x) in the form of Eq. (3.197), both assumed to satisfy Eq. (3.196), i.e.,

∫ p(x)f_{i}(x) dx = ∫ q(x)f_{i}(x) dx = c_{i} (i = 1, ⋯, m)  (3.201)

We then consider H(q(x)) to be compared to H(p(x)):

H(q) = − ∫ q(x) log q(x) dx = − ∫ q(x) log (\frac{q(x)}{p(x)}p(x)) dx
= −D_{KL}(q||p) − ∫ q(x) log p(s) dx ≤ − ∫ q(x) log p(s) dx
= − ∫ q(x) (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) dx = − ∫ p(x) (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) dx
= − ∫ p(x) log p(x) dx = H(p)  (3.202)

Here we have used the fact D_{KL}(q||p) ≥ 0 and the result in Eq. (3.201). This result again indicates that the distribution with maximal H(p) and satisfying Eq. (3.196) has to take the form in Eq. (3.197). The specific values of the m + 1 constants λ_{0}, ⋯, λ_{m} can be found by solving the m + 1 equations including Eq. (3.196) and ∫_{−∞}^{∞} p(x) dx = 1.

3.3 Bayesian Inference

3.3.1 Bayes’ Theorem

Bayesian inference is widely used in various machine learning algorithms, and it is based on Bayes’ theorem:

p(A|B) = \frac{p(A, B)}{p(B)} = \frac{p(B|A) p(A)}{p(B)}  (3.203)

If {A_{1}, ⋯, A_{n}} is a partition of sample space A, then

P (B) = ∑_{i=1}^{n} P (B|A_{i}) P (A_{i}), P (A_{i}|B) = \frac{P (B|A_{i}) P (A_{i})}{P (B)} = \frac{P (B|A_{i}) P (A_{i})}{∑_{i=1}^{n} P (B|A_{i}) P (A_{i})} (i = 1, ⋯, n)

(3.204)

For example, students take one of n elective courses {A_{1}, ⋯, A_{n}} each with probability P (A_{i}). The probability to get a B grade in course A_{i} is known to be P (B|A_{i}). Given that a student gets a B, the probability for him/her to have taken course A_{i} is P (A_{i}|B).

The method of Bayesian inference deduces a hypothesis H, a model of the probability distribution of an observed dataset D, in terms of the form and/or parameters of the distribution. The method of Bayesian inference can be an iterative process, that updates the model parameters as more data becomes available. Based on Bayes’ theorem, the Bayesian inference can be formulated in words:

posterior = \frac{likelihood × prior}{evidence}  (3.205)

or in mathematics:

p(H|D) = \frac{p(D|H) p(H)}{p(D)} ∝ p(D|H) p(H)  (3.206)

where p(H) is the proior probability of H based on some prior knowledge without observation of any data, p(H|D) is the posterior probability of H based on the observation of some data D, p(D|H) is the likelihood of H, proportional to the probability of observing D given H, and p(D) is the probability of observing data D, which is independent of H in question, and typically ignored.

Specifically, the data D and hypothesis H take the following forms:

• Data D: a collection of data points in a d-dimensional feature space, represented in a data matrix X = [x_{1}, ⋯, x_{N}] containing a set of N independent samples of a random vector x = [x_{1}, ⋯, x_{d}]^{T}, randomly and unformly drown from its probability density distribution (independent and identically distributed, or i.i.d.).

• Hypothesis H: a specific probabilistic model p(x|θ) for the dataset, parameterized by θ_{1}, ⋯, θ_{M} represented as a vector θ = [θ_{1}, ⋯, θ_{M}]^{T}, to be estimated based on the observed dataset X.

Now Bayes inference can be formulated to estimate the model parameter θ given the observed dataset X, i.e., Eq. (3.206) can be written as:

p(θ|X) = \frac{p(X|θ) p(θ)}{p(X)} = \frac{p(X|θ) p(θ)}{∑_{θ} p(X|θ) p(θ)} ∝ p(X|θ) p(θ)  (3.207)

where

• p(θ): the a priori probability, or the simply prior, of parameter θ based on some prior knowledge before any data X are observed.

• L(θ|X) ∝ p(X|θ): the likelihood of the statistical model parameters in θ (hypothesis), given the observed data X, representing how probable X is for different values of θ. If the data samples in X = [x_{1}, ⋯, x_{N}] are independently drawn from the same pdf, i.e., they are independent and identically distributed (i.i.d.), then

p(X|θ) = ∏_{n=1}^{N} p(x_{n}|θ)  (3.208)

• p(X) = ∑_{θ} p(X|θ)p(θ): the distribution of the data while the paraeter θ is marginalized. As p(X) is a constant independent of the parameter θ in question, and it is needed only for normalizing p(X|θ) p(θ), it is often dropped in the process of maximization for convenience.

• p(θ|X): the a posteriori probability, or simply the posterior of the parameter θ, after data X are observed.

There are two methods typically used for the estimation of the parameters in θ based on the observed data X:

• Maximum Likelihood Estimation (MLE) seeks to find θ that maximizes the likelihood:

θ_{MLE} = arg max_{θ} p(X|θ)  (3.209)

• Maximum A Posteriori (MAP) seeks to find θ that maximizes the posterior:

θ_{MAP} = arg max_{θ} p(θ|X) ∝ p(X|θ) p(θ)  (3.210)

The difference between the two methods is whether certain prior knowledge is available regarding the distribution p(θ) of the parameters before observing any data. If no such prior knowledge is available, i.e., all possible values of the parameters are equally likely with p(θ) = costant (uniform distribution), then the two methods will produce the same result. On the other hand, if we have some prior knowledge that the parameters obey certain distribution p(θ), e.g., they are normally distributed with certain mean and covariance, then the MAP method may produce more accurate estimation.

When we maximize either the likelihood or the posterior as a function of θ, we can equivalently maximize the logarithm (a monotonic function) of the function for mathematical convenience, the two methods above can therefore be expressed as

• MLE:

θ_{MLE} = arg max_{θ} log p(X|θ) = arg max_{θ} log (∏_{n=1}^{N} p(x_{n}|θ))

= arg max_{θ} (∑_{n=1}^{N} log p(x_{n}|θ))  (3.211)

• MAP:

θ_{MAP} = arg max_{θ} log (p(X|θ) p(θ))
= arg max_{θ} (∑_{n=1}^{N} log p(x_{n}|θ) + log p(θ))  (3.212)

Both MLE and MAP seek to find some specific values for the parameters in θ that maximize certain function of the parameters, such as the likelihood or the posterior of θ, or their logarithm. In certain situations, we can instead find the values of such parameters in θ that maximizes the expectation of the function to be maximized, now denoted by Q(θ), by the method called expectationmaximization (EM), in an iteration of two steps:

• The E-step: find the expectation Q(θ) based on the previous parameter values θ_{n};

• The M-step: find the updated parameter values θ_{n+1} that maximize the expectation Q(θ).

3.3.2 Maximum Likelihood Estimation of Model Parameters

Given an observed dataset X = [x_{1}, ⋯, x_{N}] containing N independent and identially distributed (i.i.d.) sample data points, we can build a statistical model p(x|θ), of which the parameter θ containing a set of model parameters is to be estimated based on the observed dataset X by method such as maximum likelihood estimation (MLE).

Specifically, we let θ′ be the current estimation of the true model parameter θ, and find its likelihood and log-likelihood:

L(θ′|x) ∝ p(x|θ′),  l(θ′|x) = log p(x|θ′)  (3.213)

We further consider the expectation of th log-likelihood with respect to the true distribution p(x|θ) is

E_{θ}[l(θ′|x)] = ∫ p(x|θ)l(θ′|x)dx = ∫ p(x|θ) log p(x|θ′)dx
= ∫ p(x|θ) log (p(x|θ)\frac{p(x|θ′)}{p(x|θ)}) dx
= ∫ p(x|θ) log p(x|θ)dx − ∫ p(x|θ) log \frac{p(x|θ)}{p(x|θ)′} dx
= −H(p(x|θ)) − D_{KL}(p(x|θ))||p(x|θ′))  (3.214)

where H(p(x|θ)) is the entropy of p(x|θ)), which is independent of the estimated parameter θ)′, and D_{KL}(p(x|θ))||p(x|θ′)) is the KL-divergence between p(x|θ)) and p(x|θ′)), which is zero when the estimated paramter becomes the same as the true parameter θ′ = θ.

Now we see that the optimal model parameter that maximizes the likelihood is also the one that minimizes the KL-divergence:

θ_{MLE} = arg max_{θ′} L(θ′|x) = arg min_{θ′} D_{KL}(p(x|θ))||p(x|θ′))  (3.215)

Consider the simple example of the MLE method applied to estimate the two parameters in θ = {m, Σ} of the multivariate Gaussian distribution

N (x, m, Σ) = \frac{1}{(2π)^{n/2} |Σ|^{1/2}} exp [−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)]  (3.216)

Given an observed dataset containing N i.i.d. data samples X = {x_{1}, ⋯, x_{N}}, we can get the likelihood function

L(θ|X) = L(m, Σ|X) = p(X|m, Σ) = ∏_{i=1}^{N} N (x_{i}|m, Σ)  (3.217)

and the log-likelihood function:

l(m, Σ|X) = log L(m, Σ|X) = log [∏_{i=1}^{N} N (x_{i}|m, Σ)] = ∑_{i=1}^{N} log N (x_{i}|m, Σ)

= ∑_{i=1}^{N} [−\frac{d}{2} log 2π − \frac{1}{2} log |Σ| − \frac{1}{2}(x_{i} − m)^{T} Σ^{−1}(x_{i} − m)] (3.218)

To find the optimal parameters m and Σ that maximize this log-likelihood, we set its derivative with respect to m and Σ to zero and solve the resulting equations:

• Find m

\frac{∂}{∂m}l(m, Σ|X) = −\frac{1}{2} ∑_{i=1}^{N} \frac{∂}{∂m} ((x_{i} − m)^{T} Σ^{−1}(x_{i} − m))

= −\frac{1}{2} ∑_{i=1}^{N} Σ^{−1}(x_{i} − m) = 0  (3.219)

Pre-multiplying −2Σ on both sides, we get

∑_{i=1}^{N}(x_{i} − m) = 0  (3.220)

solving which we get the optimal estimation of m:

m̂ = \frac{1}{N} ∑_{i=1}^{N} x_{i}  (3.221)

• Find Σ

\frac{∂}{∂Σ}l(m, Σ|X) = −\frac{N}{2} \frac{∂}{∂Σ} log |Σ| − \frac{1}{2} ∑_{i=1}^{N} \frac{∂}{∂Σ} ((x_{i} − m)^{T} Σ^{−1}(x_{i} − m))

= −\frac{N}{2} Σ^{−1} + \frac{1}{2} ∑_{i=1}^{N} Σ^{−1}(x_{i} − m)(x_{i} − m)^{T} Σ^{−1} = 0 (3.222)

Pre and post multiplying Σ on both sides and solving the resulting equation we get the optimal estimation of Σ:

Σ̂ = \frac{1}{N} ∑_{i=1}^{N}(x_{i} − m)(x_{i} − m)^{T}  (3.223)

Here we have used the following facts:

\frac{d}{dA} log |A| = (A^{−1})^{T} , \frac{d}{dA} (a^{T} A^{−1}b) = −(A^{−1})^{T} ab^{T} (A^{−1})^{T}

(3.224)

Note that the rank of Σ̂ in Eq. (3.223) is N − 1, representing its degrees of freedom due to the N samples, assumed to be independent, and the extra constraint in Eq. (3.221). When d > N − 1, the matrix Σ̂ does not have a full rank and is therefore non-invertible.

For more details regarding derivatives of a scalar function with respect to

     vector and matrix variables, see Matrix Cookbook here: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf.

3.3.3 Natural gradient

The gradient method is commenly used in optimization to minimize or maximize an objective function J(x) in Euclidean space based on the following iteration:

x_{n+1} = x_{n} + Δx^{∗}  (3.225)

where Δx^{∗} is the optimal increment among all those Δx = x_{n+1} − x_{n} within a small neighborhood of the current x_{n}:

Δx^{∗} = arg max_{||Δx||<C} J(x + Δx)  (3.226)

The solution of this constrained optimization problem can be found by maximizing the following Lagrangian function

L(Δx) = L(x + Δx) + μ(||Δx||^{2} − C)  (3.227)

The optimal Δx^{∗} can be found by setting the derivative of L(Δx) to zero

\frac{d}{d Δx}L(x + Δx) = g_{L}(x) + \frac{μ}{2}Δx = 0  (3.228)

and solving the resulting equation to get

Δx^{∗} = −\frac{2}{μ}g_{J}(x)  (3.229)

Now the optimal x^{∗} can be found iteratively

x_{n+1} = x_{n} + Δx_{n} = x_{n} − λg_{J}(x_{n})  (3.230)

where the step size λ = 2/μ can be adjusted as needed.

However, in some cases the function to be optimized is not in the Euclidean space spanned by a set of orthonormal basis vector in which an inner product is defined and the Euclidean distance is used to measure the difference between two points in the space, but in a Riemannian manifold (a smooth manifold locally similar to a Euclidean space), in which a small neighboring area of a point can be approximated by its tangent space spanned by a set of vectors {e_{1}, ⋯, e_{n}} not necessarily orthogonal to each other, and the squared length of an infinitesimal arc dx = [dx_{1}, ⋯, dx_{d}]^{T} on the manifold can be expressed in quadratic form as:

||dx||^{2} = (∑_{i=1}^{n} dx_{i}e_{i})^{T} (∑_{j=1}^{n} dx_{j}e_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} dx_{i}dx_{j}e_{i}^{T} e_{j}
= dx^{T} G dx  (3.231)

Here G, called the metric tensor, is a positive-definite matrix based on the inner products of the basis vectors in the tangent space:

G = [ \begin{matrix}e_{1}^{T} e_{1} & ⋯ & e_{1}^{T} e_{n} \\ ⋮ & ... & ⋮ \\ e_{n}^{T} e_{1} & ⋯ & e_{n}^{T} e_{n} \end{matrix} ]  (3.232)

In the special case when the Riemannian space is a Euclidean space spanned by orthonormal basis vectors satisfying e_{i}^{T} e_{j} = δ_{ij}, we have G = I, and the squared arc length above becomes the conventional Euclidean squared norm based on inner product: ||dx||^{2} = dx^{T} dx = ∑_{i} dx_{i}^{2}.

To minimize an objective function J(x) in a Riemannian manifold iteratively, we can still find the optimal increment step Δx as given in Eq. (3.226), that maximizes the Lagragian in Eq. (3.227), but now the squared norm is given in Eq. (3.231):

L(Δx) = L(x + Δx) + μ(Δx^{T} GΔx − C)  (3.233)

Now we solve the equation

\frac{d}{d Δx}L(x + Δx) = \frac{d}{d Δx}L(x + Δx) + μ\frac{d}{d Δx} (Δx^{T} GΔx − C)
= g_{L}(x) + \frac{μ}{2}GΔx  (3.234)

to get the optimal increment:

Δx^{∗} = −\frac{μ}{2}G^{−1}g_{L}(x) = −\frac{μ}{2}g̃_{L}(x)  (3.235)

where g̃_{L}(x) = G^{−1}g_{L}(x) is the natural gradient, and the iteration becomes

x_{n+1} = x_{n} + Δx_{n} = x_{n} − δg̃_{L}(x)  (3.236)

3.3.4 Fisher Information and Natural Gradient

The Fisher information measures the amount of information carried by an observed random x about the unknown parameter θ of a model distribution p(x|θ) of x.

The likelihood function of the parameter θ for a model p(x|θ) given observed data x is

L(θ|x) ∝ p(x|θ)  (3.237)

and the log-likelihood is

l(θ|x) = log L(θ|x) = log p(x|θ)  (3.238)

The gradient of the log-likelihood is

g_{l}(θ) = \frac{d}{dθ} log p(x|θ) = \frac{d p(x|θ)/dθ}{p(x|θ)} = \frac{g_{p}(θ)}{p(x|θ)}  (3.239)

where g_{p}(θ) = d p(x|θ)/dθ is the gradient of p(x|θ). The expectation of g_{l} is zero:

E_{θ}[g_{l}(θ)] = ∫ (\frac{g_{p}(θ)}{p(x|θ)}) p(x|θ)dx = ∫ g_{p}(θ)dx
= ∫ \frac{d}{dθ} p(x|θ)dx = \frac{d}{dθ} ∫ p(x|θ)dx = \frac{d}{dθ} 1 = 0 (3.240)

and the covariance of g_{l} is defined as the Fisher information:

F(θ) = E_{θ}[g_{l}(θ)g_{l}^{T} (θ)]  (3.241)

which can be estimated based on an observed dataset X = [x_{1}, ⋯, x_{N}]:

F(θ) = \frac{1}{N} ∑_{n=1}^{N} g_{l}(θ)g_{l}^{T} (θ)  (3.242)

The Hessian of the log-likelihood l(θ|x) is:

H_{l}(θ) = \frac{d^{2}}{dθ^{2}} l(θ|x) = \frac{d}{dθ} g_{l}(θ) = \frac{d}{dθ} [\frac{g_{p}(θ)}{p(x|θ)}]
= \frac{H_{p}(θ)p(x|θ) − g_{p}(θ)g_{p}^{T} (θ)}{p^{2}(x|θ)} = \frac{H_{p}(θ)}{p(x|θ)} − g_{l}(θ)g_{l}^{T} (θ) (3.243)

The expectation of H_{l} is

E_{θ}[H_{l}(θ)] = E_{θ} [\frac{H_{p}(θ)}{p(x|θ)}] − E_{θ} [g_{l}(θ)g_{l}^{T} (θ)]  (3.244)

The first term is

E_{θ} [\frac{H_{p}(θ)}{p(x|θ)}] = ∫ [\frac{H_{p}(θ)}{p(x|θ)}] p(x|θ)dx = ∫ H_{p}(θ)dx = ∫ \frac{d^{2}}{dθ^{2}} p(x|θ)dx
= \frac{d^{2}}{dθ^{2}} ∫ p(x|θ)dx = \frac{d^{2}}{dθ^{2}} 1 = 0  (3.245)

and the second term is the Fisher information defined in Eq. (3.241), and we have:

F(θ) = −E_{θ}[H_{l}(θ)] = E_{θ} [g_{l}(θ)g_{l}^{T} (θ)]  (3.246)

of which the component in the ith row and jth column is expected product of the ith and jth components of g_{l}(θ):

F_{ij} = E[g_{i} g_{j}] = E [\frac{∂ log p(x|θ)}{∂θ_{i}} \frac{∂ log p(x|θ)}{∂θ_{j}}]  (3.247)

We note that g_{i} = ∂ log p(x|θ)/∂θ_{i} represents how much variation in the pdf p(x|θ) of x will be caused by parameter θ_{i}, or how much information about θ_{i} is carried by x. If in the extreme case when g_{i} = 0, then x carries no information about θ_{i} as it is irrelevant to θ_{i}. Therefore the Fisher information containing all F_{ij} = E[g_{i} g_{j}] represents the amount of information about θ carried by x.

While building a statistical model p(x|θ) of the observed dataset, we need to approximate the unknown parameter θ by the estimated θ′ based on the samples of variable x, so that the KL-divergence between p(x|θ′) and the true p(x|θ) (Eq. (3.191)) is minimized:

D_{KL}(p(x|θ)||p(x|θ′)) = E_{θ}[log p(x|θ) − log p(x|θ′)]  (3.248)

Consider its first and second order derivatives with respect to θ′:

• The gradient:

g_{KL}(θ′) = E_{θ} [\frac{d}{dθ′} [log p(x|θ) − log p(x|θ′)]]
= −E_{θ} [\frac{d}{dθ′} log p(x|θ′)] = −E_{θ}[g_{l}(θ′)]  (3.249)

• The Hessian:

H_{KL}(θ′) = −E_{θ} [\frac{d}{dθ′} g_{l}(θ′)] = −E_{θ}[H_{l}(θ′)]  (3.250)

Evaluating both g_{KL}(θ′) and H_{KL}(θ′) at θ′ = θ, we get (Eqs. (3.240) and (3.246)):

g_{KL}(θ) = −E_{θ}[g_{l}(θ)] = 0,  and H_{KL}(θ) = F(θ)  (3.251)

Now the KL-divergence above can be approximated by the first three terms of its Taylor expansion in the neighborhood of θ in terms of Δθ = θ′ − θ:

D_{KL}(p(x|θ)||p(x|θ′)) ≈ D_{KL}(p(x|θ)||p(x|θ)) + g_{KL}^{T}(θ)Δθ
+ \frac{1}{2}Δθ^{T} H_{KL}(θ)Δθ = \frac{1}{2}Δθ^{T} F(θ)Δθ (3.252)

Comparing this with Eq. (3.231), we see that we can treat each component g_{i} = ∂ log p(x|θ)/∂θ_{j} of g_{l}(θ) as a basis vector in the Riemanniian space for all distributions p(x|θ), then the KL-divergence above becomes the squared distance between the two distributions:

D_{KL}(p(x|θ)||p(x|θ′)) ≈ \frac{1}{2}Δθ^{T} F(θ)Δθ = \frac{1}{2} ∑_{i} ∑_{j} Δθ_{i}Δθ_{j}g_{i}g_{j}  (3.253)

Now the optimal model parameter θ that maximizes the likelihood or equivalently minimizes the KL-divergence (Eq. (3.215)) can be found iteratively by the natural gradient method (Eq (3.236)):

θ_{n+1} = θ_{n} + Δθ_{n} = θ_{n} + δg̃_{L}(θ_{n})  (3.254)

3.4 Bias-Variance Tradeoff

The mean squared error as defined below is a common way to measure the error of the estimation of some parameter, or, more generally, the performance of a supervised learning algorithm (for either regression or classification). Let θ̂ be an estimator of the parameter θ of a data model. Then mean squared error (MSE) of this estimator is

MSE(θ̂) = E[(θ̂ − θ)^{2}]
= E[(θ̂ − E(θ̂) + E(θ̂) − θ)^{2}]
= E[(θ̂ − E(θ̂))^{2}] + 2E[(θ̂ − E(θ̂))(E(θ̂) − θ)] + E[(E(θ̂) − θ)^{2}]
= E[(θ̂ − E(θ̂))^{2}] + E[(E(θ̂) − θ)^{2}]
= Var(θ̂) + Bias(θ̂)  (3.255)

Note that the middle term disappears as

E[θ̂ − E(θ̂)] = E(θ̂) − E(θ̂) = 0  (3.256)

The equation above indicates that the MSE is composed of two parts:

• variance error Var(θ̂) = E[(θ̂ − E(θ̂))^{2}]:

caused by the oversensitivity of a learning algorithm to small fluctuations in the training dataset due to random noise rather than the intended property of the data, a problem called overfitting.

• bias error Bias(θ̂) = E[(E(θ̂) − θ)^{2}]:

caused by the under-sensitivity of a learning algorithm to the dominanting variation in the training dataset representing the essential relationship in the training dataset, a problem called underfitting.

In genral, a proper bias-variance tradeoff needs to be made for an algorithm to avoid either overfitting or underfitting, so that it can learn the essential relationship in the data while not affected by the innevitable noise in the data.

3.5 Principal Component Analysis (PCA)

Previously we discussed converting a random vector x of zero mean m_{x} = 0 and identity covariance Σ_{x} = I to y = Wx + m with desired m and Σ in Eq. (3.123) by the affine transformation. Here we consider the opposite operation to convert any random x with m_{x} = 0 and Σ_{x} to y with Σ_{y} = I, so that the components in y are completely decorrelated with zero covariance σ_{ij}^{2} = δ_{ij}. This general process, called whitening, can be easily realized also by the Cholesky decomposition of Σ_{x}:

Σ_{x}^{−1} = W^{T} W, i.e. Σ_{x} = (W^{T} W)^{−1}  (3.257)

Then the resulting vector by the whitening transform y = Wx will have the same zero mean

m_{y} = E[y] = E[Wx] = WE[x] = Wm_{x} = 0  (3.258)

but a whitened covariance:

Σ_{y} = E[yy^{T}] = E[(Wx) (Wx)^{T}] = WE[xx^{T}]W^{T} = WΣ_{x}W^{T}
= W(W^{T} W)^{−1}W^{T} = WW^{−1} (W^{T})^{−1}W^{T} = I  (3.259)

There exist infinite number of matrices W all satisfying Eq. (3.257), such as those found by the Cholesky decomposition and eigenvalue decomposition of the covariance matrix Σ_{x}, as discussed before. Here we revisit the method based on eigenvalue decomposition and show it is optimal in some sense.

consider the eigenequations of the covariance matrix Σ of an N-D random vector x = [x_{1}, ⋯, x_{N}]^{T}, now treated as N samples of a time signal:

Σ_{x}ϕ_{n} = λ_{n}ϕ_{n} (n = 1, ⋯, N)  (3.260)

As Σ is symmetric and positive semi-definite, its eigenvalues are real and nonnegative, and the corresponding eigenvectors are orthogonal to each other:

λ_{N} ≥ ⋯ ≥ λ_{1} ≥ 0, ϕ_{i}^{T} ϕ_{j} = δ_{ij} = \{ \begin{matrix}0 & i ≥ j \\ 1 & i = j \end{matrix} (i, j = 1, ⋯, N)

(3.261)

i.e., Φ^{T} = Φ^{−1} is an orthogonal matrix. The eigenequations above can be expressed in matrix form:

ΣΦ = ΦΛ, or Φ^{−1}ΣΦ = Φ^{T} ΣΦ = Λ  (3.262)

where

Λ = diag(λ_{1}, ⋯, λ_{N}), Φ = [ϕ_{1}, ⋯, ϕ_{N}]  (3.263)

are the eigenvalue and eigenvector matrix.

Further consider the following orthogonal transform of x, calledKarhunen-Loeve transform (KLT), Hotelling Transform or eigenvector Transform based on the eigenvector matrix Φ:

y = [ y_{1} ; ⋮ ; y_{N} ] = Φ^{T} x = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{N}^{T} ] x  (3.264)

where the nth component of y is y_{n} = ϕ_{n}^{T} x is the projection of x onto the nth eigenvector ϕ_{n}. Left multiplying both sides by Φ, we get the inverse KLT:

x = Φy = [ϕ_{1}, ⋯, ϕ_{N}] [ y_{1} ; ⋮ ; y_{N} ] = ∑_{n=1}^{N} y_{n}ϕ_{n}  (3.265)

We see that x is expressed as a linear combination of ϕ_{1}, ⋯, ϕ_{N} treated as an orthogonal basis that spans the N-D space. The KLT has two most favorable properties:

1. It completely decorrelate the signal 2. It maximally compact the signal energy

To see the first property, we consider covariance matrix of y:

Σ_{y} = Φ^{T} Σ_{x}Φ = Φ^{−1}Σ_{x}Φ = Λ  (3.266)

i.e.,

[ \begin{matrix}σ_{11}^{2} & ⋯ & σ_{1N}^{2} \\ ⋮ & ... & ⋮ \\ σ_{N1}^{2} & ⋯ & σ_{NN}^{2} \end{matrix} ] = [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{N} \end{matrix} ]  (3.267)

The covariance of y_{i} and y_{j} of the random vector y = Φ^{T} x is

σ_{ij}^{2} = \{ \begin{matrix}λ_{i} & i = j \\ 0 & i ≠ j \end{matrix}  (3.268)

As σ_{ij}^{2} = 0 for any i ≠ j, the components of y are completely decorrelized. after the KLT.

We next show that the KLT redistributes the dynamic energy contained in the signal so that it is maximally compacted into any M < N components, while conserving the total energy. Let A = [a_{1}, ⋯, a_{N}] be an arbitrary orthogonal matrix, based on which an orthogonal transform of x can be defined as

y = [ y_{1} ; ⋮ ; y_{N} ] = A^{T} x = [ a_{1}^{T} ; ⋮ ; a_{N}^{T} ] x  (3.269)

where y_{i} = a_{i}^{T} x. The inverse transform is:

x = Ay = [a_{1}, ⋯, a_{N}] [ y_{1} ; ⋮ ; y_{N} ] = ∑_{i=1}^{N} y_{i}a_{i}  (3.270)

As shown before, Σ_{y} = A^{T} Σ_{x}A and

trΣ_{y} = ∑_{i=1}^{N} σ_{y_{i}}^{2} = ∑_{i=1}^{N} σ_{x_{i}}^{2} = trΣ_{x}  (3.271)

i.e., the total dynamic energy (or information) contained in the signal x is conserved by the orthogonal transform. However, the energy distribution among all N components can be very different before and after the KLT transform. To see this, define the energy contained in the first M < N components after the transform y = A^{T} x, treated as an function of the transform matrix A:

ε_{M}(A) = ∑_{i=1}^{M} σ_{y_{i}}^{2} = ∑_{i=1}^{M} a_{i}^{T} Σ_{x}a_{i}  (3.272)

The optimal transform matrix A that maximizes ε_{M}(A) can be found by solving the following constrained maximization problem:

\{ \begin{matrix}maximize: ε_{M}(A) = ∑_{i=1}^{M} a_{i}^{T} Σ_{x}a_{i} \\ subject to: a_{j}^{T} a_{j} = 1 (j = 1, ⋯, N) \end{matrix}  (3.273)

The constraint a_{j}^{T} a_{j} = 1 is to guarantee that the column vectors in A are normalized. Based on the Lagrange multiplier method, we let:

\frac{∂}{∂a_{i}} [S_{M}(A) − ∑_{j=1}^{M} λ_{j}(a_{j}^{T} a_{j} − 1)] = \frac{∂}{∂a_{i}} [∑_{j=1}^{M}(a_{j}^{T} Σ_{x}a_{j} − λ_{j}a_{j}^{T} a_{j} + λ_{j})]

= \frac{∂}{∂a_{i}} (a_{i}^{T} Σ_{x}a_{i} − λ_{i}a_{i}^{T} a_{i}) =^{∗} 2 (Σ_{x}a_{i} − λ_{i}a_{i}) = 0, (i = 1, ⋯, N) (3.274)

For this to hold, a_{i} must be an eigen-vector ϕ_{i} of Σ_{x}, and we have thus proved that the optimal transform is indeed KLT. To maximize ε_{M}(A), we must have A = Φ:

ε_{M}(A) = ε_{M}(Φ) = ∑_{i=1}^{M} ϕ_{i}^{T} Σ_{x}ϕ_{i} = ∑_{i=1}^{M} λ_{i}  (3.275)

To maximize this, ϕ_{i}’s need to be the eigenvectors of Σ_{x} corresponding to M greatest eigenvalues of the N: λ_{1} ≥ ⋯ ≥ λ_{M} ≥ ⋯ ≥ λ_{N}.

Due to KLT’s properties of signal decorrelation and energy compaction, it is widely used for data compression by reducing the dimensionality of the data set.

3.6 Student’s t-test

Let x be a random variable with a normal distribution N (μ, σ^{2}), and {x_{1}, ⋯, x_{N}} be a set of N independent and identially distributed (i.i.d.) ssamples drawn from this distribution. Then the true mean of x can be estimated by its sample mean:

x̄ = \frac{1}{N} ∑_{i=1}^{N} x_{i} ∼ N (μ, σ^{2}/N)  (3.276)

i.e., x̄ is a normally distributed random variable with the same mean μ as that of the original distribution and variance σ^{2}/N, which is the same as that of the original distribution if N = 1, but approaches zero when N ⟶ ∞. We can further define another random variable z as below with a standard normal distribution (of zero mean and unit variance):

z = \frac{x̄ − μ}{σ/\sqrt{N}} ∼ N (0, 1)  (3.277)

If the standard deviation σ is not known, we can replace it by the estimated sample standard deviation based on the estimated sample variance:

S^{2} = \frac{1}{N − 1} ∑_{i=1}^{N} e_{i}^{2} = \frac{1}{N − 1} ∑_{i=1}^{N}(x_{i} − x̄)^{2} = \frac{1}{N − 1} [∑_{i=1}^{N} x_{i} − Nx̄] (3.278)

Now random variable Z can be replaced by another random variable T based on S instead of σ:

t = \frac{x̄ − μ}{S/\sqrt{N}} ∼ T_{ν}(t) = \frac{Γ (\frac{ν+1}{2})}{\sqrt{νπ}Γ (\frac{ν}{2} )} (1 + \frac{t^{2}}{ν} )^{−(ν+1)/2}  (3.279)

where Γ(n) = (n − 1)! is the gamma function. This is a random variable with Student’s t-distribution of ν = N − 1 degrees of freedom. Specially

T_{1}(t) = \frac{1}{π(1 + t^{2})}, T_{2}(t) = \frac{1}{(2 + t^{2})^{3/2}} , T_{3}(t) = \frac{6\sqrt{3}}{π(3 + t^{2})^{2}} , T_{∞}(t) = \frac{1}{\sqrt{2π}} e^{t^{2}/2}

(3.280)

When ν → ∞, the t-distribution becomes the normal distribution (the blue curve in the plots).

The confidence interval of the mean μ is defined as the interval between two limiting values such that the probability for μ to fall inside the range is (1 − α)100%:

P (L ≤ μ ≤ U) = (1 − α)100%  (3.281)

where α is the significant level. For example, if α = 0.05, then P (L ≤ μ ≤ U) = 95%.

To find the limits of such a confidence interval corresponding to a given α, we first consider the interval [−z_{α/2}, z_{α/2}] so that

P (−z_{α/2} ≤ z ≤ z_{α/2}) = ∫_{z_{−α/2}}^{z_{α/2}} N (z, 0, 1) dz = 1 − α,  (3.282)

i.e.,

∫_{−∞}^{z_{−α/2}} N (z, 0, 1) dz = ∫_{z_{α/2}}^{∞} N (z, 0, 1) dz = α/2  (3.283)

For example, if α = 0.05, we can find z_{α/2} = 1.96 from the z-Table (https://math.arizona.edu/~rsims/ma464/standardnormaltable.pdf.)

Now we can further find the lower and upper limits of the confidence interval for μ by noting that the following events are equivalent:

P (−z_{α/2} ≤ z ≤ z_{α/2}) = P (−z_{α/2} ≤ \frac{x̄ − μ}{σ/\sqrt{N}} ≤ z_{α/2})
= P (x̄ − z_{α/2} \frac{σ}{\sqrt{N}} ≤ μ ≤ x̄ + z_{α/2} \frac{σ}{\sqrt{N}} ) = 1 − α (3.284)

i.e., the lower and upper limits of the confidence interval for μ corresponding to 1 − α is:

L = x̄ − z_{α/2} \frac{σ}{\sqrt{N}} , U = x̄ + z_{α/2} \frac{σ}{\sqrt{N}}  (3.285)

so that

P (L ≤ μ ≤ U) = 1 − α  (3.286)

The standard error of the estimated mean is defined as σ/\sqrt{N}, and the upper and lower limits of the confidence interval given above can be expressed as:

x̄ ± z_{α/2} \frac{σ}{\sqrt{N}} = x̄ ± z_{α/2}SE  (3.287)

When σ is not known, it can be approximated by the estimated standard error ESE = S/\sqrt{N} based on the estimated sample standard deviation S. We can follow the same derivation above, but replacing z with standard normal distribution by t with t-distribution, and find the limits of the confidence interval as:

x̄ ± t_{α/2} \frac{S}{\sqrt{N}} = x̄ ± t_{α/2}ESE  (3.288)

where t_{α/2} are the limits satisfying the following:

P (−t_{α/2} ≤ t ≤ t_{α/2}) = ∫_{t_{−α/2}}^{t_{α/2}} T_{ν}(t)dt = 1 − α  (3.289)

which can be found by the t-Table (http://www.sjsu.edu/faculty/gerstman/StatPrimer/t-table.pdf,

given a significant level α and the degrees of freedom ν.

For example, assume x̄ is obtained based on N = 11 samples:

x̄ = \frac{1}{11} ∑_{i=1}^{11} x_{i}  (3.290)

then the degrees of freedom is ν = N − 1 = 10. For a significant level α = 0.05, we can find t_{0.05/2} = 2.228 from the t-Table. Note that this value is different from t_{0.05/2} = 1.96. However, when the sample size N becomes larger, T_{ν}(t) = T_{N−1}(t) will approach to the standard normal distribution N(z, 0, 1), and t_{α/2} ⟶ z_{α/2}.

Confidence interval for linear regression....

3.7 Hypergeometric Distribution

If n samples in a population of N samples have certain property of interest while the rest do not, the probability of the event that k samples in a subset of K samples of the population have the property is

P (k) = \begin{matrix}\binom{n}{k} & \binom{N−n}{K−k} \\ ( & {N \atop K} & ) \end{matrix},  (k = 0, ⋯, K)  (3.291)

where \binom{n}{k} = n!/(k! (n − k)! is the number of ways to choose k samples out of n samples. We have

∑_{k=0}^{K} P (k) = 1  (3.292)

We are interested in finding out whether the ratio of the samples with the property out of all samples in the subset is significantly different from the same ration of the total population, and if so, whether it is significantly higher or lower. Here the term significance can be intuitively understood as the occurance of a highly unlikely event with very low probability (e.g., 5% = 0.05).

We first define the p-value as

p(k) = ∑_{i=k}^{K} P (i) = 1 − ∑_{i=0}^{k−1} P (i)  (3.293)

based on which

• p < 0.05: the ratio of the subset is significantly higher than that of the total population

• p > 0.95: the ratio of the subset is significantly lower than that of the total population

• 0.05 < p < 0.95: the ratio of the subset is not significantly different from that of the total population

Example: In a class of N = 100 students of which K = 45 are female, n = 15 students get grade A, of which k are female. Determine whether the number of female students who get an A is significantly lower or higher than the total population of the class based on the p-value p(k) of the following values:

• p(6) = 0.9625¿0.95, significantly lower

• p(7) = 0.8963 < 0.95, not significantly different

• p(12) = 0.1047 > 0.05, not significantly different

• p(13) = 0.0394 < 0.05, significantly higher

Part II Regression 4 Linear Regression

4.1 Regression as Optimization

The goal of regression analysis is to model the relatiionship between a dependent variable y and a set of independent variables (predictors) as components of a vector x = [x_{1}, ⋯, x_{d}]^{T}, a point in the d-dimensional space, given a set of observed sample points in the training set D = {(x_{n}, y_{n}), (n = 1, ⋯, N)}, in which each sample x_{n} of the independent variables is labeled by the value of the corresponding dependent variable y_{n}. Alternatively, the training set can also be denoted by an D = {X, y} in terms of a d × N matrix X = [x_{1}, ⋯, x_{N}] and an N-D vector y = [y_{1}, ⋯, y_{N}]^{T}.

Specifically, the model is a regression function ŷ = f(x, θ), which can be either deterministic or probabilistic depending on the specific algorithm. Here θ is a vector containing a set of parameters as its components. Typically the regression function is hypothesized to take certain form (e.g., linear, polynomial, or exponential) based on some prior knowledge, while the parameters in θ are to be determined by the regression algorithm based on the training set D, so that the predicted value ŷ by the model function matches the ground truth y optimally is certain sense, as discussed below.

Geometrically, the regression function ŷ = f(x, θ) represents a hypersurface when d > 2 as generally assumed, otherwise a surface if d = 2, or a curve if d = 1). The goal to match the model to the training set is to fit the hypersurface to all training samples as points in the d + 1 dimensional space (spanned by the dependent variable y as well as the d independent variables as components in x) optimally in certain sense.

Regression analysis can also be interpreted as system modeling/identification, when the independent variable x and the dependent variable y are treated respectively as the input (stimuli) and output (responses) of a system, and the behavior of the system in terms of the relationship between such input and output is modeled by the regression function y = f(x).

While the regression algorithm tries to fit the regression function to the dataset, it should not be affected by the inevitable observation noise in the data but still catches the essential characteristics present in the data. In other words, the regression analysis should neither overfit (overly affected by noise) nor underfit (not matching valid data sufficiently closely) the observed data. This issue of making a proper tradeoff betwwn overfitting and underfitting is of great importance in all supervised learning methods in general, to be addressed repeatedly in the future.

Regression analysis can be considered as a process of supervised learning during which the regression algorithm finds the unknown model parameters in θ by learning from the training set D. It is similar to a class of supervised learning methods in pattern recognition/classification, also formulated in terms of the independent variables x treated as a set of features that characterize some patterns or opjects of interest, and the corresponding dependent variable y treated as the categorical labeling indicating to which of a set of K classes {C_{1}, ⋯, C_{K}} a pattern x belongs. In this case, the modeling process for the relationship between x and y becomes pattern classification or recognition, to be discussed in Chapter ??.

Due to this similarity between the two different types of supervised learning methods for regression and classification, all following discussion in this section for regression algorithms also applies in general to the classification algorithms to be considered in the future chapters.

The problem of regression analysis can be addressed based on different philosophical viewpoints. From the frequentist point of view, the unknown model parameters in θ are fixed deterministic variables and their values are to be estimated based on the training set. A typical method based on this viewpoint is the least squares (LS) method for both linear and nonlinear regressions.

Alternatively, from the Bayesian inference point of view (Section ??), the model parameters in θ are random variables. Their prior probability distribution p(θ) before any data are observed can be estimated based on some prior knowledge. If no such prior knowledge is available, p(w) can be simply a uniform distribution, i.e., all possible values of w are equally likely. Once the data samples in the training set X are available and observed, the posterior probability of the model parameters can be estimated based on Bayes’ theorem:

p(θ|X) = \frac{p(X|θ)p(θ)}{p(X)} ∝ p(X|w)p(θ)  (4.1)

The posterior p(θ|X) is in general a narrower distribution than the prior p(θ) due to the additional information gained from the observed data, and it is therefore a more accurate description of the model parameters. The output ŷ = f(x, θ) of such a regression model based Bayesian inference is no longer a deterministic value, but a random variable described by its probability distribution in terms of its mean and variance. Typical methods based on this point of view include Bayesian linear regression, logistic regression and Gaussian process regression.

Based on either of these two viewpoints, different methods for regression can be used to find the parameters θ for the model function ŷ = f(x, θ), such as those listed below.

• Least squares (LS) method:

This frequentist method measures how well the regression function models the training data by the residual defined as r(θ) = y − ŷ(θ), defined as the difference between the model prediction ŷ(θ) = f(x, θ) and the ground truth labeling value y corresponding to x, for each of the N data points in the observed data:

r(θ) = [ r_{1} ; ⋮ ; r_{N} ] = [ y_{1} − ŷ_{1} ; ⋮ ; y_{N} − ŷ_{N} ] = [ y_{1} − f(x_{1}, θ) ; ⋮ ; y_{N} − f(x_{N}, θ) ] = y − ŷ (4.2)

Ideally, for the residual to be zero, r(θ) = 0, we find the M model parameters as components of θ by solving this nonlinear equation system of N equations of M unknowns in θ. However, as in general N ≫ M, this system is overconstrained without a solution. We therefore can only use least squares method to find the optimal solution that minimizes the sum of squared error (SSE):

ε(θ) = \frac{1}{2}||r||^{2} = \frac{1}{2}r^{T} r = \frac{1}{2} ∑_{n=1}^{N} r_{n}^{2} = \frac{1}{2} ∑_{n=1}^{N}(y_{n} − f(x_{n}, θ))^{2}  (4.3)

Here the coefficent 1/2 is included for mathematical convenience. When divided by the total number of samples N, this SSE becomes the average of all N squared errors, and the error becomes the mean squared error (MSE). Minimizing either SEE or MSE produces the same results.

• Maximum likelihood estimate (MLE):

This Bayesian inference method measures how well the regression function models the training data in terms of the likelihood L(θ|D) of the model parameter θ based on the observed dataset D = {X, y}, which is proportional to the conditional probability of D given θ:

L(θ|D) ∝ p(D|θ) = p(y|X, θ)  (4.4)

The optimal model parameters in θ can be found as those that maximize L(θ|D) or equivalently p(y|X, θ).

We assume the residual r = y − f(x, θ) is a zero-mean random variable (without loss of generality) with a normal probability density function (pdf):

p(r) = p(y − f(x, θ)) = N (r, 0, σ^{2}) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y − f(x, θ))^{2}}{2σ^{2}} )
= N (y, f(x, θ), σ^{2}) = p(y|x, θ)  (4.5)

The justification for this assumed normal pdf is that it has the greatest entropy (Section 3.2.1), or maximum uncertainty among all possible pdfs with the same variance, i.e., given σ^{2}, the assumed normal pdf imposes least amount of constraint and thereby minimum bias.

The zero-mean pdf of r = y − f(x, θ) can also be considered as the pdf of y with mean f(x, θ), the conditional pdf p(y|x, θ) of y given θ as well as x, based on the assumption that y and x are related by y = f(x, θ). Then we can find the likelihood of θ given the data samples in the training set, all assumed to be independent and identically distributed (i.i.d.):

L(θ|D) = p(y|X, θ) = ∏_{n=1}^{N} p(y = y_{n}|x_{n}, θ)

= ∏_{n=1}^{N} \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y_{n} − f(x_{n}, θ))^{2}}{2σ^{2}} )  (4.6)

or, equivalently and for mathematical convenience, minimize the negative log likelihood function:

−l(θ|D) = − log L(θ|X, y) = − ∑_{n=1}^{N} log p(y = y_{n}|x_{n}, θ)  (4.7)

= − ∑_{n=1}^{N} log [\frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y_{n} − f(x, θ))^{2}}{2σ^{2}} )]

= \frac{N}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∑_{n=1}^{N} ||r||^{2}  (4.8)

The first term and the coefficient 1/σ^{2} in the second term are both independent of θ and can therefore be dropped. We note that this result is equivalent to that in Eq. (4.3), i.e., we can find the optimal θ by either maximizing the likelihood L(θ|D), or equivalently minimizing the SSE ε(θ).

We further note that negative log likelihood −l(θ|D) = − log L(θ|X, y) in the equation above can also be interpreted differently as the cross entropy (Section 3.186) between an unknown ground truth probability P and the model probability Q:

H(P, Q) = − ∑_{i} P_{i} log Q_{i} = −E_{P} [log Q]  (4.9)

We rewrite Eq. (4.7) as the average of the logarithm of the model distribution Q = p(y|x, θ) over all N observed samples in the dataset D = {X, y}:

−l(θ|D) = − ∑_{n=1}^{N} log p(y = y_{n}|x_{n}, θ) = −E_{(X,y)} [log p(y = y_{n}|x_{n}, θ)]

(4.10)

and realize that this is the cross entropy H(P, Q) estimated based on the N samples, which can be minimized for model Q to match the ground truth P as closely as possible.

• Maximum A Posteriori (MAP)

This is also a Bayesian inference method that measures how well the regression function models the training data by the posterior probability of the parameter θ given in Eq. (4.1), proportional to the product of the likelihood L(θ|D) ∝ p(D|θ) and the prior p(θ). If no prior knowledge about θ is available, then p(θ) is a uniform distribution, then MAP is equivalent to MLE. However, if certain prior knowledge regarding θ does exist, and the prior is not uniform, then the posterior better characterizes θ than the likelihood and MAP may produce better result than MLE.

We conclude that regression analysis, and in general many supervised learning algorithms to be considered in the following chapters, can be treated as an over-constrained optimization problem with many more observed data points in the raining set than the number of unknown parameters in the model. Following different philosorphical view points, this optimization problem can be addressed differently, such as minimizing the sum of squared errors, maximizing the likelihood of the model parameters, or minimizing the negative likelihood, i.e., the cross entropy between the model and unknown ground truth distributions of the given dataset, to be discussed in details in the following sections.

More generally, some of the concepts and methods in regression analysis discussed in the following sections are also shared by many supervised learning algorithms for classification. For example, the general method of neural network is conceptually based on some ideas very different from regression, but both of these two methods can be described mathematically as a function y = f(x, θ) that models the relationship between the independent variable x and dependent variable y. While this function is called the regress function in this chapter, it is also a way to describe the complex behavior of a multilayer deep learning network to be discussed in Chapter ??

4.2 Linear Least Squares Regression

In linear regression, the relationship between the dependent variable y and d independent variables in x = [x_{1}, ⋯, x_{d}]^{T} is modeled by a linear hypothesis function:

ŷ = f(x, w) = b + ∑_{i=1}^{d} w_{i}x_{i} = ∑_{i=0}^{d} w_{i}x_{i} = x^{T} w  (4.11)

where x = [x_{0} = 1, x_{1}, ⋯, x_{d}]^{T} is redefined as an augmented d + 1 dimensional vector containing x_{0} = 1 as well as the d independent variables, and w = [w_{0} = b, w_{1}, ⋯, w_{d}]^{T} is an argmented d + 1 dimensional vector containing w_{0} = b as well as the d weights as the parameters, to be determined based on the given dataset.

Geometrically, this linear regression function f(x) = w^{T} x is a straight line if d = 1, a plane if d = 2, or a hyperplane if d > 2, in the d + 1 dimensional space spanned by y as well as {x_{1}, ⋯, x_{d}}. The function has an intercept f(0) = w_{0} = b along the y dimension and a normal direction [w_{1}, ⋯, w_{d}, −1]^{T}. When thresholded by the plane y = 0, the regression function becomes an equation f(x) = x^{T} w = 0, representing a point if d = 1, a straight line if d = 2, a plane or hyperplane if d ≥ 3, in the d-dimensional space spanned by {x_{1}, ⋯, x_{d}}, as shown in the figure below for d = 2.

We denote by X the matrix containing the N augmented data points x_{n} = [x_{0n} = 1, x_{1n}, ⋯, x_{dn}]^{T} (n = 1, ⋯, N) as its column vectors

X = [ \begin{matrix}1 & ⋯ & 1 \\ x_{11} & ⋯ & x_{1N} \\ ⋮ & ... & ⋮ \\ x_{d1} & ⋯ & x_{dN} \end{matrix} ]_{(d+1)×N} = [x_{1}, ⋯, x_{N}]  (4.12)

and its transpose can be written as:

X^{T} = [ \begin{matrix}1 & x_{11} & ⋯ & x_{d1} \\ ⋮ & ⋮ & ... & ⋮ \\ 1 & x_{1N} & ⋯ & x_{dN} \end{matrix} ]_{N×(d+1)} = [1, \underline{x}_{1}, ⋯, \underline{x}_{d}]  (4.13)

where 1 = [1, ⋯, 1]^{T}, and \underline{x}_{i} = [x_{i1}, ⋯, x_{iN}]^{T} (i = 1, ⋯, d) is an Ndimensional vectors containing the ith components of all N data vectors in the dataset.

Now the linear regression problem is simply to find the model parameter, here the weight vector w, so that the model prediction

ŷ = X^{T} w  (4.14)

matches the ground truth labeling y. In other words, the residual of the linear model defined as r = y − X^{T} w need to be zero:

r = y − X^{T} w = 0  (4.15)

This is a linear system containing N equations of M unknowns.

If the number of data samples is equal to the number of unknown parameters, i.e., N = d+1, then X is a square and invertible matrix (assuming all N samples are independent and therefore X has a full rank), and the equation above can be solved to get the desired weight vector:

w = (X^{T})^{−1}y  (4.16)

However, as typically N ≫ d+1, i.e., there are many more data samples than the unknown parameters, X is non-invertible and X^{T} w = y is over-determined linear system without a solution, i.e, the prediction ŷ = X^{T} w can never match the ground truth labeling y. We need to find an optimal solution w^{∗} that minimizes the SSE (proportional to MSE):

ε(w) = \frac{1}{2} ∑_{n=1}^{B} r_{n}^{2} = \frac{1}{2}||r||^{2} = \frac{1}{2}||y − X^{T} w||^{2}
= \frac{1}{2}y^{T} y − y^{T} X^{T} w + \frac{1}{2}w^{T} XX^{T} w  (4.17)

To find the optimal w that minimizes ε(w), we set the gradient of ε to zero and get the normal equation:

g_{ε}(w) = \frac{d}{dw} ε(w) = XX^{T} w − Xy = 0, i.e. XX^{T} w = Xy  (4.18)

Solving for w, we get the optimal weight vector:

w^{∗} = (XX^{T})^{−1}Xy = (X^{T})^{−}y  (4.19)

where (X^{T})^{−} = (XX^{T})^{−1}X is the (d + 1) × N pseudo-inverse (Section A.15) of the N × (d + 1) matrix X^{T}. This method for estimating the regression model parameters is called ordinary least squares (OLS).

Having found the parameter w for the linear regression model ŷ = x^{T} w, we can predict the outputs corresponding to any unlabed test dataset X_{∗} = [x_{1∗}, ⋯, x_{M∗}]:

ŷ_{∗} = f_{∗} = X_{∗}^{T} w^{∗} = [1, \underline{x}_{1}, ⋯, \underline{x}_{N}] [ w_{0}^{∗} ; w_{1}^{∗} ; ⋮ ; w_{d}^{∗} ] = w_{0}^{∗}1 + ∑_{i=1}^{d} w_{i}^{∗}\underline{x}_{i}  (4.20)

which is a linear combination of 1 and \{\underline{x}_{1}, ⋯, \underline{x}_{d}\}

We further consider the evaluation of the linear regression result in terms of how well it fits the training data. We first rewrite the normal equation in Eq. (4.18) as:

XX^{T} w − Xy = X (X^{T} w − y) = X (ŷ − y)

= Xr = [ \begin{matrix}1 & ⋯ & 1 \\ x_{11} & ⋯ & x_{1N} \\ ⋮ & ... & ⋮ \\ x_{d1} & ⋯ & x_{dN} \end{matrix} ] r = [ 1^{T} ; \underline{x}_{1}^{T} ; ⋮ ; \underline{x}_{d}^{T} ] r = 0  (4.21)

The last equality indicate that the residual vector r is parpendicular to each of the d + 1 vectors \{1, \underline{x}_{1}, ⋯, \underline{x}_{d}\}:

1^{T} r = ∑_{n=1}^{N} r_{n} = 0, \underline{x}_{1}^{T} r = ⋯ = \underline{x}_{N}^{T}r = 0  (4.22)

We see that the residual r = y − ŷ = y − X^{T} w^{∗} based on the optimal w^{∗} represents the shortest distance between y and any point in the space spanned by \{1, \underline{x}_{1}, ⋯, \underline{x}_{d}\}, i.e., any other estimate ŷ′ = X^{T} w also in the space but based on a non-optmal w ≠ w^{∗} must have a greater residual r′ = y − ŷ′ > r, i.e., w^{∗} is indeed the optimal parameter based on which the sum of quared error ||r||^{2} of the model ŷ = X^{T} w^{∗} is minimized.

We further consider some quantitative measurement for evaluating how well the linear regression model y = w^{T} x fits the given dataset. To do so, we first derive some properties of the model based on its output ŷ = X^{T} w = X^{T} (XX^{T})^{−1}Xy:

||ŷ||^{2} = ŷ^{T} ŷ = (X^{T} (XX^{T})^{−1}Xy)^{T} (X^{T} (XX^{T})^{−1}Xy)
= (y^{T} X^{T} (XX^{T})^{−1}X) (X^{T} (XX^{T})^{−1}Xy)
= y^{T} X^{T} (XX^{T})^{−1}Xy = y^{T} ŷ  (4.23)

based on which we also get

r^{T} ŷ = (y − ŷ)^{T} ŷ = y^{T} ŷ − ŷ^{T} ŷ = 0  (4.24)
||r||^{2} = r^{T} r = (y − ŷ)^{T} (y − ŷ) = y^{T} y − 2y^{T} ŷ + ŷ^{T} ŷ
= y^{T} y − y^{T} ŷ = y^{T} (y − ŷ) = y^{T} r  (4.25)

and

||ŷ||^{2} + ||r||^{2} = y^{T} ŷ + y^{T} r = y^{T} (ŷ + r) = y^{T} y = ||y||^{2}  (4.26)

We further find the mean of {y_{1}, ⋯, y_{N}}

ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n} = \frac{1}{N} y^{T} 1  (4.27)

and define a vector ȳ = ȳ 1. The three vectors y − ŷ, y − ȳ and ŷ − ȳ are all perpendicular to vector 1:

(y − ȳ)^{T} 1 = ∑_{n=1}^{N}(y_{n} − ȳ) = N(ȳ − ȳ) = 0

(y − ŷ)^{T} 1 = ∑_{n=1}^{N}(y_{n} − ŷ_{n}) = ∑_{n=1}^{N} r_{n} = 0 Eq.(4.22)
(ŷ − ȳ)^{T} 1 = (ŷ − y + y − ȳ)^{T} 1 = (ŷ − y)^{T} 1 + (y − ȳ)^{T} 1 = 0 (4.28)

Based on these three vectors, we further define the following sums of squares:

• The total sum of squares (TSS) for the total variation in data:

T SS = ||y − ȳ||^{2} = ∑_{n=1}^{N}(y_{n} − ȳ)^{2}  (4.29)

• The explained sum of squares (ESS) for the variation of data explained by the regression model:

ESS = ||ŷ − ȳ||^{2} = ∑_{n=1}^{N}(ŷ_{n} − ȳ)^{2}  (4.30)

• Residual sum of squares (RSS) for the variation of data not explained by the data (due to noise or discrepancy between the data and the model):

RSS = ||r||^{2} = ||y − ŷ||^{2} = ∑_{n=1}^{N}(y_{n} − ŷ_{n})^{2}  (4.31)

We can show that the total sum of squares is the sum of the explained sum of squares and the residual sum of squares:

T SS = ||y − ȳ||^{2} = ||y − ȳ + ŷ − ŷ||^{2} = ||(ŷ − ȳ) + (y − ŷ)||^{2}
= ||ŷ − ȳ||^{2} + 2(y − ŷ)^{T} (ŷ − ȳ) + ||y − ŷ||^{2}
= ||ŷ − ȳ||^{2} + 2r^{T} (ŷ − ȳ) + ||r||^{2}
= ESS + 2r^{T} ŷ − 2r^{T} ȳ + RSS = ESS + RSS  (4.32)

The last equality is due to the fact that the two middle terms are both zero:

\{ \begin{matrix}r^{T} ȳ = ∑_{n=1}^{N} r_{n}ȳ = ȳ∑_{n=1}^{N} r_{n} = 0 Eq.(4.22), \\ r^{T} ŷ = (y − ŷ)^{T} ŷ = y^{T} ŷ − ŷ^{T} ŷ = 0 Eq.(4.24) \end{matrix}  (4.33)

We can now measure the goodness of the model by the coefficient of determination, denoted by R^{2} (R-squared), defined as the percentage of variance explained by the model:

R^{2} = \frac{ESS}{T SS} = 1 − \frac{RSS}{T SS}  (4.34)

Given the total sum of squares TSS, if the model residual RSS is small, then ESS is large, indicating most of the variation in the data can be explained by the model, and R^{2} is large, i.e., the model is a good fit of the data.

In the special case of 1-dimensional dataset {(x_{n}, y_{n}) (n = 1, ⋯, N)}, how closely the two variables x and y are correlated to each other can be measured by correlation coefficient defined as:

ρ = \frac{σ_{xy}^{2}}{\sqrt{σ_{x}^{2} σ_{y}^{2}}} = \frac{σ_{xy}^{2}}{σ_{x} σ_{y}}  (4.35)

where

x̄ = \frac{1}{N} ∑_{n=1}^{N} x_{n}, ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n}

and

σ_{x}^{2} = \frac{1}{N} ∑_{n=1}^{N} x_{n}^{2} − x̄^{2}, σ_{y}^{2} = \frac{1}{N} ∑_{n=1}^{N} y_{n}^{2} − ȳ^{2}, σ_{xy}^{2} = \frac{1}{N} ∑_{n=1}^{N} x_{n}y_{n} − x̄ȳ

We can further find the linear regression model y = w_{0} + w_{1} x in terms of the model parameters:

w = [ \begin{matrix}w_{0} \\ w_{1} \end{matrix} ] = (XX^{T})^{−1}Xy

= ([ \begin{matrix}1 & ⋯ & 1 \\ x_{1} & ⋯ & x_{N} \end{matrix} ] [ \begin{matrix}1 & x_{1} \\ ⋮ & ⋮ \\ 1 & x_{N} \end{matrix} ])^{−1} [ \begin{matrix}1 & ⋯ & 1 \\ x_{1} & ⋯ & x_{N} \end{matrix} ] [ y_{1} ; ⋮ ; y_{N} ]

= [ \begin{matrix}N & ∑_{n=1}^{N} x_{n} \\ ∑_{n=1}^{N} x_{n} & ∑_{n=1}^{N} x_{n}^{2} \end{matrix} ]^{−1} [ \begin{matrix}∑_{n=1}^{N} y_{n} \\ ∑_{n=1}^{N} x_{n}y_{n} \end{matrix} ] = [ \begin{matrix}1 & x̄ \\ x̄ & σ_{x}^{2} + x̄^{2} \end{matrix} ]^{−1} [ ȳ ; σ_{xy}^{2} + x̄ȳ ]
= \frac{1}{σ_{x}^{2}} [ \begin{matrix}σ_{x}^{2} + x̄^{2} & −x̄ \\ −x̄ & 1 \end{matrix} ] [ ȳ ; σ_{xy}^{2} + x̄ȳ ] = [ \begin{matrix}ȳ − σ_{xy}^{2}/σ_{x}^{2} x̄ \\ σ_{xy}^{2}/σ_{x}^{2} \end{matrix} ]  (4.36)

Based on these model parameters w_{1} = σ_{xy}^{2}/σ_{x}^{2} and w_{0} = ȳ − x̄w_{1}, we get the linear regression model:

ŷ = w_{0} + w_{1}x = ȳ − \frac{σ_{xy}^{2}}{σ_{x}^{2}} x̄ + \frac{σ_{xy}^{2}}{σ_{x}^{2}} x or  \frac{σ_{xy}^{2}}{σ_{x}^{2}} = \frac{y − ȳ}{x − x̄}  (4.37)

The coefficient of determination R^{2} for the goodness of the regression is closely related to the correlation coefficient ρ for correlation of the given data (but independent of the model), as one would intuitively expect. Consider

\frac{1}{N} RSS = \frac{1}{N} ||y − ŷ||^{2} = \frac{1}{N} ∑_{n=1}^{N}(y_{n} − w_{0} − w_{1}x_{n})^{2}

= \frac{1}{N} ∑_{n=1}^{N}(y_{n} − (ȳ − x̄w_{1}) − w_{1}x_{n})^{2} = \frac{1}{N} ∑_{n=1}^{N}[(y_{n} − ȳ) − w_{1}(x_{n} − x̄)]^{2}

= \frac{1}{N} ∑_{n=1}^{N}[(y_{n} − ȳ)^{2} − 2w_{1}(y_{n} − ȳ)(x_{n} − x̄) + w_{1}^{2}(x_{n} − x̄)^{2}]

= σ_{y}^{2} − 2w_{1}σ_{xy}^{2} + w_{1}^{2}σ_{x}^{2} = σ_{y}^{2} − 2\frac{σ_{xy}^{2}}{σ_{x}^{2}} σ_{xy}^{2} + \frac{σ_{xy}^{4}}{σ_{x}^{4}} σ_{x}^{2} = σ_{y}^{2} − \frac{σ_{xy}^{4}}{σ_{x}^{2}}
= σ_{y}^{2} (1 − \frac{σ_{xy}^{4}}{σ_{x}^{2}σ_{y}^{2}} ) = σ_{y}^{2}(1 − ρ^{2})  (4.38)

We see that if the two variables x and y are highly correlated and ρ is close to 1, then the error of the model is small, i.e., RSS = ||r||^{2} is small and R^{2} is large, indicating the LS linear model fits the data well.

Example 1:

Find a linear regression function y = w_{0} + w_{1} x to fit the following N = 6 points:

\begin{matrix}x & −3.4 & −2.1 & −0.8 & 0.3 & 1.7 & 2.5 \\ y & −0.76 & −1.04 & 1.75 & 1.82 & 3.17 & 3.15 \end{matrix}

Based on the data, we get

ȳ = 1.348, σ_{x}^{2} = 4.22, σ_{y}^{2} = 2.85, σ_{xy}^{2} = 3.27, ρ = \frac{σ_{xy}^{2}}{σ_{x}σ_{y}} = 0.94

The augmented data array is:

X = [ \begin{matrix}1 & 1 & 1 & 1 & 1 & 1 \\ −3.4 & −2.1 & −0.8 & 0.3 & 1.7 & 2.5 \end{matrix} ]

Solving the over-constrained equation system

y = [ \begin{matrix}−0.76 \\ −1.04 \\ 1.75 \\ 1.82 \\ 3.17 \\ 3.15 \end{matrix} ] = X^{T} w = [ \begin{matrix}1 & −3.4 \\ 1 & −2.1 \\ 1 & −0.8 \\ 1 & 0.3 \\ 1 & 1.7 \\ 1 & 2.5 \end{matrix} ] [ \begin{matrix}w_{0} \\ w_{1} \end{matrix} ]

we get

w = X^{−}y = (XX^{T})^{−1}X y = [ \begin{matrix}0.13 & 0.15 & 0.16 & 0.17 & 0.19 & 0.20 \\ −0.12 & −0.07 & −0.02 & 0.02 & 0.08 & 0.11 \end{matrix} ] [ \begin{matrix}−0.76 \\ −1.04 \\ 1.75 \\ 1.82 \\ 3.17 \\ 3.15 \end{matrix} ] = [ \begin{matrix}1.58 \\ 0.78 \end{matrix} ]

and the linear regression model function y = w_{0} + w_{1} x = 1.58 + 0.78 x, a straight line with intercept w_{0} = 1.58, slope w_{1} = 0.78, and normal direction [0.78, −1]^{T}. The LS error is ||r|| = ||y − ŷ|| = 1.38. We can further get ȳ = [1.348, 1.348, 1.348, 1.348, 1.348, 1.348]^{T}, ŷ = X^{T} w = [−1.054, −0.047, 0.961, 1.813, 2.898, 3.518]^{T}, and

ESS = 15.21, RSS = 1.91, T SS = 17.12, R^{2} = 0.89

We can also check that

\frac{1}{N} RSS = σ_{y}^{2}(1 − ρ^{2}) = 0.318

Example 2:

\begin{matrix}x & −3.0 & −2.5 & −2.0 & −1.5 & −1.0 & −0.5 & 0.0 & 0.5 & 1.0 & 1.5 & 2.0 & 2.5 & 3.0 \\ y_{1} & −2.67 & −3.76 & −0.64 & −0.93 & −0.01 & −1.42 & 3.38 & −0.47 & 1.77 & 3.67 & 6.40 & 4.49 & 8.82 \\ y_{2} & 2.73 & −9.38 & −1.44 & −9.86 & −2.82 & −7.29 & 8.31 & 0.98 & 10.18 & 8.58 & 10.86 & 0.77 & 0.20 \end{matrix}

w_{0} w_{1} ESS RSS T SS R^{2}  ρ
dataset1 1.43 1.72 134.71 30.18 164.89 0.817 0.904
dataset2 0.91 1.92 166.98 443.92 610.89 0.273 0.523

Example 3:

The figure below shows a set of N = 30 data points {(x_{i}, y_{i}), i = 1, ⋯, 30} fitted by a 1-D linear regression function y = w_{0} + w_{1}x, a straight line, with w_{0} = 1.81, w_{1} = 3.22. The correlation coefficient is ρ = 0.98, and ESS = 734.78, RSS = 32.98, T SS = 767.77, R^{2} = 0.96.

Example 4:

A set of N = 90 data points {(x_{i}, y_{i}), i = 1, ⋯, 90} fitted by a 2-D linear regression function y = w_{0}+w_{1}x_{1}+w_{2}x_{2}, with w_{0} = 2.82, w_{1} = 1.85, w_{2} = 4.31. This is a plane as shown in the figure below from two perspective view angles. Here ESS = 677.00, RSS = 124.48, T SS = 802.47 and R^{2} = 0.84.

Finally we note that regression analysis can be considered as a binary classification, when the regression function y = f(x), as a hypersurface in a d + 1 dimensional space spanned by {x_{1}, ⋯, x_{d}, y}, is thresholded by a constant C (e.g., C = 0). The resulting equation f(x) = C defines a hypersurface in the d dimensional space spanned by {x_{1}, ⋯, x_{d}}, which partitions the space into two parts in such a way that all points on one side of the hypersurface satisfy f(x) > C, while all points on the other side satisfy f(x) < C. In other words, the regression function is a binary classifier that separates every point x in the d dimensional space into two classes C_{+} and C_{−}, depending on whether f(x) is greater or smaller than C. Now each y_{n} corresponding to x_{n} in the given dataset D = {X, y} can be treated as a label indicating x_{n} belong to class C_{+} if y_{n} > C, or C_{−} if y_{n} < C, and the regression problem becomes a binary classification problem.

4.3 Bias-Variance Tradeoff and Ridge Regression

In a generic regression problem, the goal is to model the unknown relationship y = f(x) between the dependent variable y and the d independent variables x = [x_{1}, ⋯, x_{d}]^{T}, by a regression function f̂(x), based on a set of given training data D = {(x_{n}, y_{n}) n = 1, ⋯, N} containing a set of observed data samples inevitably contaminated by some random noise e. Now corresponding to a noisy x the observed value y = f(x) + e is also random and so is the regresson model f̂(x).

We assum the random noise e has zero mean and is independent of f̂:

E[e] = 0, Var[e] = E[(e − E[e]^{2})] = E[e^{2}] = σ^{2}E[e f̂] = E[e] E[f̂] (4.39)

and measure how well the model f̂(x) fits the noisy data y = f(x) + e by the mean squared error (MSE):

E[(y − f̂)^{2}] = E[(f + e − f̂)^{2}] = E [[(f − Ef̂) + (Ef̂− f̂) + e]^{2}]
= E [(f − Ef̂)^{2} + (Ef̂− f̂)^{2} + e^{2} + 2[e(f − Ef̂) + e(Ef̂− f̂) + (f − Ef̂)(Ef̂− f̂)]]
= E[(f − Ef̂)^{2}] + E[(Ef̂− f̂)^{2}] + E(e^{2})
+ 2 E[e(f − Ef̂)] + 2 E[e(Ef̂− f̂)] + 2 E[(Ef̂− f̂)(f − Ef̂)]
= E[(f − Ef̂)^{2}] + E[(Ef̂− f̂)^{2}] + E[e^{2}]  (4.40)

The last equality is due to the fact that all three cross terms are zero:

\{ \begin{matrix}E[e(f − Ef̂)] = E[e] E[f − Ef̂] = 0 \\ E[e(Ef̂− f̂)] = E[e] E[Ef̂− f̂] = 0 \\ E[(Ef̂− f̂)(f − Ef̂)] = (Ef̂− Ef̂)(f − Ef̂) = 0 \end{matrix}  (4.41)

The three remaining terms represent three different types of error:

• Irreducible Error E[e^{2}] = σ^{2}, due to the inevitable observation error.

• Bias Error: E[(f −Ef̂)^{2}] = (f −Ef̂)^{2}, the expected squared bias, the difference between the estimated function and the true function, measuring how well the estimated function f̂ models the true function f;

• Variance Error: E[f̂− Ef̂]^{2} = Var[f̂], the variance of the estimated function f̂, measuring the variation of the model due to the noise in the training data.

We therefore get the following general relationship in terms of the three types of error:

Mean Squared Error = Variance + Bias Error + Irreducible Error (4.42)

The variance error and bias error are illustrated below:

Given the total MSE and the irreducible error σ^{2}, the bias error and the variance error are complementary, i.e., if one error is higher, the other is lower, depending on the model function f̂ based on a specific algorithm. This relationship is called the bias-variance tradoff, which is related to the major issue of overfitting vs. underfitting in supervised learning in general:

• Overfitting:

In order to match the training data closely, a complex model may be required to take into consideration detailed variations in the data which may be cuased by noise. Such a model tends to have a low bias error, but may suffer from high variance error as it may be overly affected by the random noise (possibly some outliers) and does not generalize well accross different datasets contaminated by varying noise.

• Underfitting:

In order to avoid being overly affected by the random noise, a simple model is used with the possible consequence that the variation in the data is not closely match. Such a model tends to have low variance error as it is insensitive to the varying random noise accross datasets, but it may suffer from high bias error as it may not match the valid signal variations.

These two possibilities are summarized in the table and the plot below.

Overfit Underfit

Variance error High

Low

Bias error  Low  High

(4.43)

As both overfitting and underfitting result in poor predicttion capability, they need to be avoided by making a proper tradeoff between bias-error/underfitting and variance-error/overfitting. This is an important issue not only in the context of regression, but also in general in all supervised learning algorithms, for the purpose of predicting either a value as in regression, or a categorical labeling in classification, based on the output ŷ = f(x) as a function of the given data point x. A general framework for all such algorithms is shown below, where the algorithm in the box is to come up with a model f(x, θ) of which the output ŷ is to match the labeling y of the data point x in the training set D = {X, y} so that the residual r is minimized.

The issue of overfitting versus underfitting in supervised learning is further illustrated below, for regression on the left and classification on the right.

We see that in regression, the data points are modeled by two different regression functions, while in classification, the 2-D space is partitioned into two regions corresponding to two classes by two different decision boundaries. The issue is, which of the two models, in either regression or classification, fits the data better, in terms of underfitting versus overfitting:

• Underfitting:

The simple linear regression function or decision boundary (red) may underft the data as it may miss some legitimate variations in the data.

• Overfitting:

The complex regression model, e.g., a high order polynomial, or decision boundary (blue) may overfit the data as it may be overly affected by the variation due to the observation noise.

In general, based on a single set of training data points, it is impossible to distinguish between variations due to valid signal and random noise,, and to judge whether a learning algorithm either over or under fits the data. However, based on multiple datasets contaminated by different random noise, overfitting problem can be detected if the algorithm produces significantly different results when trained by different datasets. Based on this idea, the method of crossvalidation can be used to train a learning algorithm by one subset of the training dataset and then test the algorithm by another subset of the data. if the residual error during testing is large, then the algorithm may suffer from overfitting.

Many regression and classification algorithms address the issue of overfitting versus underfitting by a process called regularization to prevent overfitting, espeicially when the problem is ill-conditioned, so that the model is less sensitive to noise in the training data and therefore more generalizable, while still able to capture the essential nature and behavior of the signal. Typically this is done by adjusting some hyperparameter of the model to make a proper tradeoff between the two extremes of over and underfitting.

As an exammple, consider the method of ridge regression for LS linear regression. If the N data points in the training set are barely independent, then matrix XX^{T} has a full rank but may be very close to singularity, i.e., its smallest eigenvalue λ_{min}, the smallest singular value σ_{min} = \sqrt{λ_{min}} of A, is close to zero, and the normal equation XX^{T} w = Xy in Eq. (4.18) is ill-conditioned. Now the solution w = (X^{T})^{−}y based on the pseudo-inverse is unstable and prone to noise. Any minor change due to noise in the dataset (in either X or y) may cause a major change in w, i.e., the resulting w has a large variance error and the regression model overfits the data and is therefore poor generalizable.

In this case, the ridge regression method can be used to regularize the illconditioned problem by constructing an objective function J(w) that contains a regularization or penalty term λ||w||^{2}/2 to discourage large weights while minimizing the error ε = ||r||^{2}/2:

J(w) = \frac{1}{2}||y − X^{T} w||^{2} + \frac{λ}{2}||w||^{2}  (4.44)

This method is in general called weight decay and the hyperparameter λ is called the weight decay parameter. Solving the equation

\frac{d}{dw} J(w) = XX^{T} w − Xy + λw = (XX^{T} + λI)w − Xy = 0  (4.45)

we get

w^{∗} = (XX^{T} + λI)^{−1}Xy  (4.46)

Now a proper tradeoff between accuracy and stability can be made by adjusting the hyperparameter λ:

• Small λ: the solution is more accurate but also more prone to noise and therefore less stable, i.e., large variance error but small bias error, or overfitting;

• Large λ: the solution is more stable as it is less affacted by noise, but it may be less accurate, i.e., small variance error but large bias error, or underfitting.

To prevent overfitting, regularization by means of including a penalty term ||w||^{2} in the objective function is commonly used in many supervised learning algorithms to discourage large values in the weight vector w as a parameter of the model for the dataset, such as in various regression algorithms in this chapter, and some popular classification algorithms in later chapters, including support vector machine in Chapter ?? and back propagation neural networks in Chapter ??.

4.4 Regression Based on Basis Functions

The method of linear regression considered previously can be generalized to model nonlinear relationships between the dependent variable y and the independent variables in x = [x_{1}, ⋯, x_{d}]^{T} by a regression function as a linear combination of K nonlinear basis functions of x:

ŷ = f(x) = ∑_{k=1}^{K} w_{k} φ_{k}(x, θ) = φ(x)^{T} w  (4.47)

where φ_{k}(x, θ), (k = 1, ⋯, K) is one of the basis function parameterized by θ that span the function space in which ŷ resides, and φ(x) = [φ_{1}(x), ⋯, φ_{K}(x)]^{T} is a K-D vector. Some of the commonly used basis functions are listed below:

• Power function:

φ_{k}(x) = x^{k−1}  (4.48)

The regression function f(x) is (K − 1)th order polynomial.

• Sinusoidal function:

φ_{k}(x, ω) = sin((k − 1)ωx), or φ_{k}(x, ω) = cos((k − 1)ωx) (4.49)

The regression function f(x) is similar to the sine or cosine series expansion of the given function.

• Gassian function:

φ_{k}(x, c_{k}, a) = exp(a||x − c_{k}||^{2})  (4.50)

This function is one member of a family of functions called radial basis functions (RBFs), all of which are centered at point c_{k}.

• Sigmoid function:

φ_{k}(x, c_{k}, a) = \frac{1}{1 + e^{−a||x−c_{k}||}}  (4.51)

These four types of basis functions are shown in the plot below for K = 7. In practice, choosing a proper type of basis functions may be a trial-and-error process depending on the specific dataset.

The original d-dimensional data space spanned by x_{1}, ⋯, x_{d} is now converted to a K-dimensional space spanned by the K basis functions φ_{1}(x), ⋯, φ_{K}(x). Applying such a regression model to each data point x_{n}, we get

ŷ_{n} = ∑_{k=1}^{K} w_{k} φ_{k}(x_{n}) = φ(x_{n})^{T} w (n = 1, ⋯, N)  (4.52)

These N equations can be written in matrix form:

ŷ = [ y_{1} ; ⋮ ; y_{N} ] = [ \begin{matrix}φ_{1}(x_{1}) & ⋯ & φ_{K}(x_{1}) \\ ⋮ & ... & ⋮ \\ φ_{1}(x_{N}) & ⋯ & φ_{K}(x_{N}) \end{matrix} ] [ w_{1} ; ⋮ ; w_{K} ]

= [ φ^{T} (x_{1}) ; ⋮ ; φ_{K}^{T}(x_{N}) ] w = Φ^{T} w  (4.53)

where Φ is a K × N matrix:

Φ = [ \begin{matrix}φ_{1}(x_{1}) & ⋯ & φ_{1}(x_{N}) \\ ⋮ & ... & ⋮ \\ φ_{K}(x_{1}) & ⋯ & φ_{K}(x_{N}) \end{matrix} ] = [φ(x_{1}), ⋯, φ(x_{N})]  (4.54)

Typically there are many more sample data points than the basis functions, i.e., N ≫ K.

If, specially, φ(u) = u is an identity function, then Eq. (4.47) becomes the same as Eq. (4.11) for linear regression. Also, as the nonlinear model ŷ = Φ^{T} w in Eq. (4.53) is in the same form as the linear model ŷ = X^{T} w in Eq. (4.11), it can be solved the same way as in Eq. (4.19), by simply replacing X by Φ:

w^{∗} = (ΦΦ^{T})^{−1}Φy = (Φ^{T})^{−}y  (4.55)

where (Φ^{T})^{−} = (ΦΦ^{T})^{−1}Φ is the K × N pseudo-inverse of the N × K matrix Φ^{T}. Now we get the output of the regression model

ŷ = Φ^{T} w = Φ^{T} Φ^{−}y = Φ^{T} (ΦΦ^{T})^{−1}Φy  (4.56)

The Matlab code segment below shows the essential part of the algorithm, where x(n) is the nth data sample x_{n} out of the total of N, y(x) is the vector y = [y_{1}, ⋯, y_{N}]^{T} with y_{n} = f(x_{n}), and c(k) contains the parameters for the kth basis function ϕ_{k}(x) = ϕ(x, c(k)) represented by phi in the code.

     for n=1:N
         for k=1:K
             Phi(k,n)=phi(x(n),c(k));
         end
     end
     w=pinv(Phi')*y(x);     % weight vector by LS method
     yhat=Phi'*w;               % reconstructed function

Example 1:

The plots below show the approximations of a given function based on a set of N samples using linear regression based on the four different types of basis functions. We see that as the number of basis functions K increases, the approximation becomes more acurrate. more accurate Example 2: The linear regression method based on basis functions is used to approximate

a 2-D function f(x) = f(x_{1}, x_{2}) shown in the figure below (left), based on the function values in {y_{1}, ⋯, y_{N}} at a set N sample points {x_{1}, ⋯, x_{N}} taken from a grid in the 2-D space (right). The regression is based on K 2-D Gaussian basis functions centered at different locations in the 2-D space shown in the following figure for K = 9.

The figure below shows the results based on different numbers of basis functions. Again, we see that the approximation becomes more accurate as the value of K increases.

In addition to the different types of basis functions considered above, we further consider the following basis functions:

φ_{k}(x, w_{k}, b_{k}) = g (w_{k}^{T} x + b_{k}) = g (∑_{i=1}^{d} w_{ki}x_{i} + b_{k}) = g (∑_{i=0}^{d} w_{ki}x_{i}) (4.57)

where g(x), called activation function, can take some different forms such as a sigmoid or rectified linear function (rectified linear unit or ReLU),

g(x) = \{ \begin{matrix}1/(1 + e^{−x}) & sigmoid function \\ max(0, x) = \{ \begin{matrix}0 & x < 0 \\ x & x ≥ 0 \end{matrix} & ReLU \end{matrix}  (4.58)

Now the regression function in Eq. (4.47) can be written as below for multiple outputs :

ŷ_{j} = ∑_{k=1}^{K} w_{jk}^{o} φ_{k}(x, θ_{k}) = ∑_{k=1}^{K} w_{jk}^{o} g (∑_{i=0}^{d} w_{ki}^{h}x_{i}) (j = 1, ⋯, m) (4.59)

which can be interpreted as the output of a three-layer neural network shown below, containing an input layer of d nodes, a middle or hidden layer of K nodes with weights w_{ki}^{h}, (i = 0, ⋯, d, k = 1, ⋯, K), and an output layer of m nodes with weights w_{jk}^{o}, (k = 1, ⋯, K, j = 1, ⋯, m). These nodes are also called neurons, mimicking the biological neurons in the brain.

As shown in figure, the nodes of the input layer take the d components of a data point x from the dataset one at a time, while each of the K nodes of the hidden layer calculates their linear combination w_{k}^{T} x, and, together with a bias term b_{k}, generates the basis function φ_{k}(x) = g (w_{k}^{T} x + b_{k}) as its output. Finally, each node of the output layer takes these K basis function from the K hidden layer and generates as its output the regression function ŷ = f(x) = ∑_{k=1}^{K} w_{k}φ_{k}(x) as the model of the dataset.

This network depends on two sets of weights, those represented by w_{ki}^{h} for the hidden nodes, and those represented by w_{jk}^{o} (found in Eq. (4.55)) for the output nodes. When there are multiple nodes in the output layer, they can each approximate a different function. This three-layer network is of particular interest in neural network algorithms, to be considered in Chapter 9.///

Example 3: This method of regression based on basis functions is used to approximate

the same 2-D function f(x) = f(x_{1}, x_{2}) in the previous example, as a linear combination of K basis functions φ_{k}(x) = w_{k}^{T} x + b_{k} with each parameterized by w_{k} and b_{k} (randomly generated). The Matlab code segment below shows the essential part of the program for this example, where f is a arbitrary function to be approxmiated.

a=@(x) x.*(x>0);        % ReLU activation function, or
a=@(x) 1/(1+exp(-x));   % Sigmoid activation function
g=@(x,w,b)a(w'*x+b);    % the basis function
[X,Y]=meshgrid(xmin:1:xmax, ymin:1:ymax); % define 2-D points
nx=size(X,2);           % number of samples in first dimension
ny=size(Y,1);           % number of samples in second dimension
N=nx*ny;                % total number of data samples
Phi=zeros(N,K);        % K basis functions evaluated at N sample points
W=1-2*rand(2,K);        % random initialization of weights
b=1-2*rand(1,K);        % and biases for K basis functions
n=0;
for i=1:nx
    x(1)=X(1,i);        % first component
    for j=1:ny
        x(2)=Y(j,1);    % second component
        n=n+1;          % of the nth sample
        Y(i,j)=func(x); % function evalued at the sample
        for k=1:K       % basis functions evaluated at the sample
            Phi(k,n)=g(x,W(:,k),b(k));
        end
    end
end
y=reshape(Y,N,1);       % convert function values to vector
w=pinv(Phi')*y;         % find weights by LS method
yhat=Phi'*w;             % reconstructed function
Yhat=reshape(yhat,m,n); % convert vector y to 2-D to compare w/ Y

The figures below show the basis functions based on both the sigmoid and ReLU activation functions, and the corresponding regression results. We note that as more basis functions are used, the more accurately the function is approximated.

• Regression based on sigmoid function:

• Regression based on ReLU function:

4.5 Bayesian Linear Regression

We revisit the linear regression problem but from the viewpoint of Bayesian inference. Now the parameter w in the linear regression model ŷ = f(x, w) = x^{T} w is assumed to be a random vector, and, as a function of w, the regression function f(x, w) is also random, and so is the residual based on the training set D = {X, y}:

r = y − ŷ = y − X^{T} w,  i.e.  y = ŷ + r = f(X) + r = X^{T} w + r (4.60)

We further assume w has a zero-mean normal prior pdf:

p(w) = N (w, 0, Σ_{w}) ∝ exp (−\frac{1}{2}w^{T} Σ_{w}^{−1}w) ,  (4.61)

based on the desire that the weights in w should take values around zero, instead of large values on either positive or negative, to avoid potiential overfitting. We also assume the residual r has a zero-mean normal pdf with a diagonal covariance matrix, i.e., all components of r are indepenent of each other:

p(r) = N (r, 0, σ_{r}^{2}I) ∝ exp (−\frac{1}{2σ_{r}^{2}} r^{T} r) = exp (−\frac{||r||^{2}}{2σ_{r}^{2}} )  (4.62)

As a linear transformation of w, the regression function ŷ = X^{T} w also has a zero-mean Gaussian distribution, and so is y = X^{T} w + r as the sum of two Gaussians.

To obtain the model parameter w, we consider its likelihood L(w|D) based on the observed data D = {X, y}:

L(w|D) ∝ p(D|w) = p(y|X, w) = p(X^{T} w + r) = N (y, X^{T} w, σ_{r}^{2}I)
∝ exp (−\frac{1}{2σ_{r}^{2}} (y − X^{T} w)^{T} (y − X^{T} w)) = exp (−\frac{1}{2σ_{r}^{2}} ||r||^{2})(4.63)

This is the same Gaussian as p(r) = N (r, 0, σ_{r}^{2}I), but shifted by X^{T} w. Now we can find the optimal model parameter w^{∗} by maximizing the likelihood L(w|D). This method is called maximum likelihood estimation (MLE). But as maximizing L(w|D) is equivalent to minimizing the SSE ||r||^{2}, the MLE method is equivalent to the least squares method considered before.

Maximizing the likelihood above is equivalent to minimizing the negative log likelihood:

−l(w|X, y) = − log L(w|X, y) = \frac{||r||^{2}}{2σ_{r}^{2}} ∝ ||r||^{2}  (4.64)

which is the same as the squared error ε(w) defined in Eq. (4.17) to be minimized by the LS method previously considered. We therefore see that the least squares method and the maximum likelihood method are equivalent to each other in this case.

We can further find the posterior p(w|D) of the model parameter w given the observed data D, as the product of prior p(w) and likelihood L(w|D) ∝ p(y|X, w):

p(w|D) = p(w|X, y) = \frac{p(D|w) p(w)}{p(D)} = \frac{p(y|X, w) p(w)}{p(y|X)}
∝ p(y|X, w) p(w) = N (y, X^{T} w, σ_{r}^{2}I) N (w, 0, Σ_{w})
∝ exp [−\frac{1}{2} (\frac{1}{σ_{r}^{2}} (y − X^{T} w)^{T} (y − X^{T} w) + w^{T} Σ_{w}^{−1}w)]
∝ exp [−\frac{1}{2} (\frac{1}{σ_{r}^{2}} y^{T} y − \frac{2}{σ_{r}^{2}} (Xy)^{T} w + \frac{1}{σ_{r}^{2}} w^{T} XX^{T} w + w^{T} Σ_{w}^{−1}w)]
∝ exp [−\frac{1}{2} (w^{T} (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1}) w − \frac{2}{σ_{r}^{2}} (Xy)^{T} w)]  (4.65)

where all constant scaling factors (irrelavant to w) are dropped for simplicity. As both the prior and the likelihood are Gaussian, this this posterior, when normalized, is also Gaussian in terms of the mean m_{w/d} and covariance Σ_{w/d}:

p(w|D) = N (w, m_{w/d}, Σ_{w/d}) ∝ exp [−\frac{1}{2} ((w − m_{w/d})^{T} Σ_{w/d}^{−1}(w − m_{w/d}))]
∝ exp [−\frac{1}{2} (w^{T} Σ_{w/d}^{−1}w − 2m_{w/d}^{T}Σ_{w/d}^{−1}w)](4.66)

with the following covariance and mean:

Σ_{w/d} = (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1})^{−1}
m_{w/d} = \frac{1}{σ_{r}^{2}} Σ_{w/d}Xy = (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy  (4.67)

Now we can find the optimal model parameter w^{∗} = m_{w/d} equal to the mean of the Gaussian posterior where it is maximized. This solution is called the maximum a posteriori (MAP) estimate of the model parameter w.

We can now predict the function value of any unlabeled test data point x_{∗} based on the conditional probability distribution p(f_{∗}|x_{∗}, D) of the regression model ŷ_{∗} = f_{∗} = f(x_{∗}) = x_{∗}^{T} w, which, as a linear transformation of the random parameters in w, is also Gaussian (Section 3.1.3).

p(f_{∗}|x_{∗}, D) = p(x_{∗}^{T} w|x_{∗}, X, y) = N (f_{∗}, x_{∗}^{T} m_{w/d}, x_{∗}^{T} Σ_{w/d}x_{∗})  (4.68)

with the following mean and variance:

E[f_{∗}|x_{∗}, X, y] = x_{∗}^{T} m_{w/d} = x_{∗}^{T} (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy  (4.69)
Var[f_{∗}|x_{∗}, X, y] = x_{∗}^{T} Σ_{w/d}x_{∗} = x_{∗}^{T} (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1})^{−1} x_{∗} (4.70)

Furthermore, given a set of M unlabeled test data points in X_{∗} = [x_{1∗}, ⋯, x_{M∗}]^{T}, we can find

ŷ_{∗} = f_{∗} = f(x_{∗}) = [ f(x_{1∗}) ; ⋮ ; f(x_{M∗}) ] = [ x_{1∗}^{T} ; ⋮ ; x_{M∗}^{T} ] w = [x_{1∗}, ⋯, x_{M∗}]^{T} w = X_{∗}^{T} w

(4.71)

As a linear combination of Guassian w, f_{∗} = X_{∗}^{T} w is also Gaussian:

p(f_{∗}|X_{∗}, D) = p(f_{∗}|X_{∗}, X, y) = N (f_{∗}, X_{∗}^{T} m_{w/d}, X_{∗}^{T} Σ_{w/d}X_{∗})  (4.72)

with the following mean and covariance:

E[f_{∗}|X_{∗}, X, y] = E[X_{∗}^{T} w] = X_{∗}^{T} E[w] = X_{∗}^{T} m_{w/d} = X_{∗}^{T} (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy (4.73)
Cov(f_{∗}|X_{∗}, X, y) = E[f_{∗}f_{∗}^{T}] = E[X_{∗}^{T} ww^{T} X_{∗}] = X_{∗}^{T} E[ww^{T}]X_{∗} = X_{∗}^{T} Σ_{w/d}X_{∗} (4.74)

The mean E[f_{∗}|X_{∗}, X, y] above can be considered as the estimated function value ŷ_{∗} = f_{∗} = X_{∗}^{T} w at the test points in X_{∗}, while the variance Var[f_{∗}|X_{∗}, X, y] as the confidence or certainty of the estimation.

Such a result produced by the Bayesian method in terms of the mean and covariance of f_{∗} = ŷ_{∗} as a random vector is quite different from the result produced by a frequentist method such as the linear least square regression in terms of an estimated value of ŷ_{∗} as a deterministic variable. However, in the special case of σ_{r} = 0, i.e., the residual r is no longer treated as a random variable, then Eq. (4.73) above becomes the same as Eq. (4.20) produced by linear least square regression, in other words, the linear method method can be considered as a special case of the Bayesian regression when the residual is deterministic.

The method considered above is based on the linear regression model f(x) = x^{T} w, but it can also be generalized to nonlinear functions, same as in the case of regression based on basis functions discussed in Section ??. To do so, the regression function is assumed to be a linear combination of a set of K nonlinear mapping functions φ_{k}(x) (k = 1, ⋯, K):

y = f(x) + e = ∑_{k=1}^{K} w_{k} φ_{k}(x) + r = w^{T} ϕ(x) + r = ϕ^{T} (x)w + r  (4.75)

where

w = [w_{1}, ⋯, w_{K}]^{T} , ϕ(x) = [φ_{1}(x), ⋯, φ_{K}(x)]^{T}  (4.76)

For example, some commenly used mapping functions are listed in Eq. (4.57).

Applying this regression model to each of the N observed data points in D = {(x_{n}, y_{n}), (n = 1, ⋯, N)}, we get ŷ_{n} = ∑_{k=1}^{K} w_{k} φ_{k}(x_{n}) with residual r_{n} = y_{n} − ŷ_{n}, which can be written in matrix form:

y = [ y_{1} ; ⋮ ; y_{N} ]_{N×1} = [ \begin{matrix}φ_{1}(x_{1}) & ⋯ & φ_{K}(x_{1}) \\ ⋮ & ... & ⋮ \\ φ_{1}(x_{N}) & ⋯ & φ_{K}(x_{N}) \end{matrix} ]_{N×K} [ w_{1} ; ⋮ ; w_{K} ]_{K×1} + [ r_{1} ; ⋮ ; r_{N} ]_{N×1}

= [ ϕ_{1}^{T} ; ⋮ ; ϕ_{N}^{T} ] w + r = Φ^{T} w + r  (4.77)

where, as previously defined in Eq. (4.54),

ϕ_{n} = ϕ(x_{n}) = [ φ_{1}(x_{n}) ; ⋮ ; φ_{K}(x_{n}) ] , Φ = [ϕ_{1}, ⋯, ϕ_{N}]  (4.78)

As this nonlinear model ŷ = Φ^{T} w is in the same form as the linear model ŷ = X^{T} w, all previous discussion for the linear Bayesian regression is still valid for this nonlinear model, if X = [x_{1}, ⋯, x_{N}]_{(d+1)×N} is replaced by Φ = [ϕ_{1}, ⋯, ϕ_{N}]_{K×N}. We can get the covariance matrix and mean vector of the posterior p(w|D) in the K-dimensional space based on the observed data D = {X, y}:

Σ_{w/d} = (\frac{1}{σ_{r}^{2}} ΦΦ^{T} + Σ_{w}^{−1})^{−1}
m_{w/d} = \frac{1}{σ_{r}^{2}} Σ_{w/d}Φy = (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Φy  (4.79)

The function ŷ_{∗} = f(X_{∗}) of the multiple test points in X_{∗} = [x_{∗1}, ⋯, x_{∗M}]^{T} is a linear transformation of w:

ŷ_{∗} = f_{∗} = f(x_{∗}) = [ f_{1}(x_{∗}) ; ⋮ ; f_{M}(x_{∗}) ] = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{M}^{T} ]_{∗} w = [ϕ_{1}, ⋯, ϕ_{M}]^{T} w = Φ_{∗}^{T} w

(4.80)

and it also has a Gaussian joint distribution:

p(f_{∗}|X_{∗}, X, y) = N (Φ_{∗}^{T} m_{w/d}, Φ_{∗}^{T} Σ_{w/d}Φ_{∗})  (4.81)

with the mean and covariance:

E[f_{∗}|X_{∗}, X, y] = E[Φ_{∗}^{T} w] = Φ_{∗}^{T} E[w] = Φ_{∗}^{T} m_{w/d}
Cov(f_{∗}|X_{∗}, X, y) = E[f_{∗}f_{∗}^{T}] = E[Φ_{∗}^{T} ww^{T} Φ_{∗}] = Φ_{∗}^{T} E[ww^{T}]Φ_{∗} = Φ_{∗}^{T} Σ_{w/d}Φ_{∗} (4.82)

They can be reformulated to become:

E[f_{∗}|X_{∗}, X, y] = Φ_{∗}^{T} m_{w/d} = Φ_{∗}^{T} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Φy
= Φ_{∗}^{T} (Σ_{w}Φ)(Σ_{w}Φ)^{−1} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} (Φ^{−1})^{−1}y
= Φ_{∗}^{T} Σ_{w}Φ [Φ^{−1} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1}) Σ_{w}Φ]^{−1} y
= Φ_{∗}^{T} Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}y  (4.83)

and

Cov(f_{∗}|X_{∗}, X, y) = Φ_{∗}^{T} Σ_{w/d}Φ_{∗} = Φ_{∗}^{T} (\frac{1}{σ_{r}^{2}} ΦΦ^{T} + Σ_{w}^{−1})^{−1} Φ_{∗}
= Φ_{∗}^{T} [Σ_{w} − Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}Φ^{T} Σ_{w}] Φ_{∗}
= Φ_{∗}^{T} Σ_{w}Φ_{∗} − Φ_{∗}^{T} Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}Φ^{T} Σ_{w}Φ_{∗}(4.84)

where we have used Woodbury identity in Section ??.

We note that in these expressions the mapping function φ(x) only shows up in the quadratic form of Φ^{T} Σ_{w}Φ, Φ^{T} Σ_{w}Φ_{∗}, or Φ_{∗}^{T} Σ_{w}Φ_{∗}. This suggests that we may consider using kernel method (Section ??, by which some significant advantage may be gained (such as in the support-vector machines (Section 13). Specifically, if we replace the mn-th component ϕ^{T} (x_{m})Σ_{w}ϕ(x_{n}) of Φ^{T} Σ_{w}Φ by a kernel function only in terms of x_{m} and x_{n}:

ϕ^{T} (x_{m})Σ_{w}ϕ(x_{n}) = k(x_{m}, x_{n}) = k(x_{n}, x_{m})  (4.85)

then the mapping function φ(x) no longer needs to be explicitly specified. This idea is called the kernel trick, Now we have

Φ^{T} Σ_{w}Φ = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{N} ] Σ_{w} [ϕ_{1}, ⋯, ϕ_{N}] = [ \begin{matrix}k(x_{1}, x_{1}) & ⋯ & k(x_{1}, x_{N}) \\ ⋮ & ... & ⋮ \\ k(x_{N}, x_{1}) & ⋯ & k(x_{N}, x_{N}) \end{matrix} ] = K

(4.86)

and similarly we also have Φ^{T} Σ_{w}Φ_{∗} = K_{∗} and Φ_{∗}^{T} Σ_{w}Φ = K_{∗}^{T}. These kernel functions are symmetric, same as all covariance matrices. Now the mean and covariance of f_{∗} given above can be expressed as:

E[f_{∗}] = K_{∗}^{T} (K + σ_{r}^{2}I)^{−1} y
Cov(f_{∗}) = K_{∗∗} − K_{∗}^{T} (K + σ_{r}^{2}I)^{−1} K_{∗}  (4.87)

As a general summary of the method discussed above, we note that it is essentially different from the frequentist approach, in the sense that the estimated result produced is no longer a set specific values of some unknown variables in question (function values, parameters of regression models, etc.), but the probability distribution of these variables in terms of the expectation and covariance. Consequently, we can not only treat the expectation as the estimated values, but also use the variance as the confidence or certainty of such an estimation.

Below is a segment of Matlab code for estimating the mean and covariance of weight vector w by the Bayes linear regression. Here X and y are for X and y of the training data D, and Xstar, Phi and Phistar are for X_{∗}, Φ and Φ_{∗}, respectively.

    SigmaP=eye(d);                  % prior variance of r in original space
    SigmaQ=eye(D);                  % prior variance of r in feature space
    Sigmaw1=inv(X*X'/sigmaN^2+inv(SigmaP));        % covariance of w by Bayes regression
    Sigmaw2=inv(Phi0*Phi0'/sigmaN^2+inv(SigmaQ));  % covariance of w with basis function
    w0=inv(X*X')*X*y                % weight vector w by LS method
    w1=Sigmaw1*X*y/sigmaN^2         % mean of w by Bayes regression
    w2=Sigmaw2*Phi0*y/sigmaN^2;     % mean of w vector by Bayes regression with basis function
    y0=w0'*Xstar;                   % y by least squares method
    y1=w1'*Xstar;                   % y by Bayes regression
    y2=w2'*Phistar0;                % y by Bayes regression with basis function
    for i=1:M
        sgm1(i)=sqrt(Xstar(:,i)'*Sigmaw1*Xstar(:,i));      % variance of estimated y
        sgm2(i)=sqrt(Phistar(:,i)'*Sigmaw2*Phistar(:,i));
    end

Example

Consider a 2-D linear function

f(x) = w^{T} x = [w_{0}, w_{1}] [ \begin{matrix}x_{0} \\ x_{1} \end{matrix} ] = w_{0}x_{0} + w_{1}x_{1} = 0.5 + 0.5 x_{1}  (4.88)

where we have assumed x_{0} = 1, w_{0} = −1/2 and w_{1} = 1/2, i.e., the function is actually a straight line f(x) = (−1 + x)/2 with intercept w_{0} = −1/2 and slope w_{1} = 1/2. The distribution of the additive noise (residual) is p(e) = N (0, σ_{r} = 0.8). The prior distributions of the weights w = [w_{0}, w_{1}]^{T} is assumed to be p(w) = N (0, I), and posterior p(w|D) can be obtained based on the observed data D = {X, y}, both shown in the figure below. We see that the posterior is concentrated around ŵ_{1} = −0.28 and ŵ_{2} = 0.39, the estimates of the true weights w_{1} = −0.5 and w_{2} = 0.5, based on the observed noisy data.

Various regression methods including both the least-squared method and the Bayesian linear regression methods, and in feature space spanned by a set of basis functions ϕ_{i}(x), (i = 1, ⋯, D = 5) based on each of the three functions given above are carried out with their results shown in the figure below:

The LS errors for these methods are listed below:

Method  LS Error w = [w_{1}, w_{2}]^{T}
Pseudo inverse 0.4730 [−0.28, 0.39]^{T}

Bayesian linear regression 0.4742 [−0.22, 0.38]^{T}

ϕ = exp(a||x − c||^{2})  0.6196
ϕ = ||x − c||  0.4429
ϕ = exp(−a||x − c||^{2})  0.4459

(4.89)

We see that the linear methods (the pseudo inverse and the Bayesian regression) produce similar estimated weights error, but smaller error can be achieved by the same method of Bayesian linear regression if proper nonlinear mapping is used (e.g., the last two mapping functions).

5 Nonlinear Regression

5.1 Nonlinear Least Squares Regression

The goal of nonlinear least squares (LS) regresion is to model the nonlinear relationship between the dependent variable y and independent variables in x = [x_{1}, ⋯, x_{d}]^{T}, based on the given training dataset D = {(x_{n}, y_{n}), n = 1, ⋯, N}, by a nonlinear regression function parameterized by a set of M parameters represented by θ = [θ_{1}, ⋯, θ_{M}]^{T}:

ŷ = f(x_{1}, ⋯, x_{d}, θ_{1}, ⋯, θ_{M}) = f(x, θ)  (5.1)

While the specific form of the nonlinear function is typically determined based on certain prior knowledge, we need to find the parameeter θ for the model to fit the training dataset D, so that, its output ŷ matches the ground truth labeling y in the training set, i.e., the residual r = y − ŷ is ideally zero:

r(θ) = [ r_{1} ; ⋮ ; r_{N} ] = [ y_{1} ; ⋮ ; y_{N} ] − [ f_{1}(θ) ; ⋮ ; f_{N}(θ) ] = y − ŷ(θ) = y − f(X, θ) = 0 (5.2)

where ŷ(x_{n}, θ) = f_{n}(θ) = f(x_{n}, θ) is the predicted output by the regression function based on the nth data point x_{n}. We see that the regression problem is equivalent to solving a nonlinear equation system of N equations for the M unknown parameters in θ. As typically there are many more data samples than the unknown parameters, i.e., N ≫ M, the equation r = 0 is over-overconstrained without an exact solution. We therefore consider the the optimal parameter θ^{∗} that minimizes the sum-of-squared error based on the residual r:

ε(θ) = \frac{1}{2}||r||^{2} = \frac{1}{2} ∑_{n=1}^{N} r_{n}^{2} = \frac{1}{2} ∑_{n=1}^{N}[y_{n} − f_{n}(θ)]^{2}  (5.3)

To do so, we first consider the gradient descent method based on the gradient of this error function:

g_{ε}(θ) = \frac{d}{dθ} ε(θ) = \frac{d}{dθ} (\frac{1}{2} ∑_{n=1}^{N} r_{n}^{2}) = ∑_{n=1}^{N} \frac{d r_{n}}{dθ} r_{n}

= − ∑_{n=1}^{N} \frac{d}{dθ} f_{n}(θ)[y_{n} − f_{n}(θ)] = −J_{f}^{T} r  (5.4)

where J_{f} is the N × M Jacobian matrix J_{f} of f(X, θ). The mth component of the gradient vector is:

g_{m}(θ) = \frac{∂}{∂θ_{m}} ε(θ) = \frac{1}{2} \frac{∂}{∂θ_{m}} ||r||^{2} = \frac{1}{2} ∑_{n=1}^{N} \frac{∂}{∂θ_{m}} r_{n}^{2} = ∑_{n=1}^{N} \frac{∂r_{n}}{∂θ_{m}} r_{n}

= ∑_{n=1}^{N} \frac{∂f_{n}(θ)}{∂θ_{m}}[f_{n}(θ) − y_{n}] = ∑_{n=1}^{N} J_{nm} [f_{n}(θ) − y_{n}] ∑_{n=1}^{N} J_{nm} r_{n}  (m = 1, ⋯, M) (5.5)

where J_{nm} = ∂ f_{n}(θ)/∂θ_{m} is the element in the nth row and mth column of J_{f}, which is simply the negative version of Jacobian of r = y − f(X, θ):

J_{r}(θ) = \frac{d}{dθ} r(θ) = \frac{d}{dθ} [y − f(X, θ)] = −\frac{d}{dθ} f(X, θ) = −J_{f}(θ)  (5.6)

The optimal parameters in θ that minimizes ε(θ) can now be found iteratively by the gradient descent method

θ_{n+1} = θ_{n} − δ_{n}g_{ε}(θ_{n}) = θ_{n} − δ_{n}J_{f}^{T} r  (5.7)

Setting the value for the step size δ_{n} properly may be tricky, as it depends on the specific landscape of the error hypersurface ε(θ). If δ_{n} is too large, a minimum may be skipped over, but if δ_{n} is too large, the convergence of the iteration may be too slow.

This gradient method is also called batch gradient descent (BGD), as the gradient in Eq. (5.4) is calculated based on the entire batch of all N data points in the dataset D in each iteration. When the size of the dataset is large, BGD may be slow due to the high computational complexity. In this case the method of stochastic gradient descent (SGD) can be used, which approximates the gradient based on only one of the N data points uniformly randomly chosen from the dataset in each iteration, i.e., the summation in Eq. (5.4) contains only one term corresponding to the chosen sample x_{n}:

g_{n}(θ) = \frac{d}{dθ} \frac{1}{2}r_{n}^{2} = r_{n} \frac{d r_{n}}{dθ}  (5.8)

The expectation of g_{n} based on one data point is the same as the true gradient (scaled by 1/N) based on all N data points in the dataset

E[g_{n}(θ)] = ∑_{n=1}^{N} P (x_{n})g_{n}(θ) = ∑_{n=1}^{N} \frac{1}{N} g_{n}(θ) = \frac{1}{N} ∑_{n=1}^{N} \frac{dr_{n}}{dθ} r_{n} = \frac{1}{N} g(θ) (5.9)

As the data are typically noisy, the SGD gradient based on only one data point may be in a direction very different from the true gradient based on the mean of all N data points, and the resulting search path in the M-D space may be in a very jagged zig-zag shape, and the iteration may converge slowly. However, as the computational complexity in each iteration is much reduced, we can afford to carry out a large number of iterations with an overall search path generally coinciding with that of the BGD method.

Some trade-off between BGD and SGD can be made so that the gradient is estimated based on more than one but less than all N data points in the dataset, the resulting method, called mini-batch gradient descent, can take advantage of both methods. The methods of stochastic and mini-batch gradient descent are widely used in machine learning, such as in the method of artificial neural networks, which can be treated as a nonlinear regression problem, considered in Chapter ??.

Alternatively, the optimal parameter θ^{∗} of the nonlinear regression function f(x, θ) can also be found by the Gauss-Newton method, which directly solves the nonlinear over-determined equation system r(θ) = y − f(X, θ) = 0 in Eq. (5.2), based on the iterative Newton-Raphson method in Eq. (1.110) in Chapter 1:

θ_{n+1} = θ_{n} − J_{r}^{−} r = θ_{n} − (J_{r}^{T} J_{r})^{−1}J_{r}^{T} r = θ_{n} + (J_{f}^{T} J_{f})^{−1}J_{f}^{T} r  (5.10)

The last equality is due to the fact that J_{r} = −J_{f}. Here J_{f}^{T} J_{f} in the pseudoinverse is actually an approximation of the Hessian H_{ε} of the error function ε(θ), as shown in the example in Chapter 2, and −J_{f}^{T} r = g_{ε}(θ) is the gradient of the error function given in Eq. (5.4). So the iteration above is essentially:

θ_{n+1} = θ_{n} − J_{r}^{−} r ≈ θ_{n} − H_{ε}^{−1}(θ_{n})g_{ε}(θ_{n})  (5.11)

same as the Newton-Raphson method based on the second order Hessian as well as the first oder gradient. We therefore see that the Gauss-Newton method is actually for minimizing the sum-of-squares error is essentially the same as the Newton-Raphson method, with the only difference that the Hessian H_{ε} ≈ J_{f}^{T} J_{f} as the second order derivative of the error function ε(θ) is approximated by the Jacobian J_{f}θ as the first order derivative of the regression function f(x, θ), making the implementation computationally easier.

In the special case of linear regression function f(w) = x^{T} w, we have:

ŷ = f(X, w) = [ f(x_{1}, w) ; ⋮ ; f(x_{N}, w) ] = [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] w = [x_{1}, ⋯, x_{N}]^{T} w = X^{T} w (5.12)

where w = [w_{0}, w_{1}, ⋯, w_{d}]^{T} and x_{i} = [1, x_{i1}, ⋯, x_{id}]^{T} are both augmented d + 1 dimensional vectors and X = [x_{1}, ⋯, x_{N}]. The Taylor series is

f(X, w + Δw) = X^{T} (w + Δw) = X^{T} w + X^{T} Δw  (5.13)

and the Jacobian is

J_{f}(w) = X^{T}  (5.14)

Now Eq. (5.10) becomes

w_{n+1} = w_{n} + (X^{T})^{−}(y − X^{T} w_{n}) = w_{n} + (XX^{T})^{−1}X(y − X^{T} w_{n})
= w_{n} + (XX^{T})^{−1}Xy − (XX^{T})^{−1}(XX^{T})w_{n}
= (XX^{T})^{−1}Xy = (X^{T})^{−}y  (5.15)

i.e., the optimal parameter w^{∗} can be found in a single step without iteration, same as that in Eq. (4.19).

The Matlab code for the essential parts of the algorithm is listed below, where data(:,1) and data(:,2) are the observed data containing N samples x_{1}, ⋯, x_{N} in the first column and their corresponding function values y_{1}, ⋯, y_{N} in the second column. In the code segment below, the regression function (e.g., an exponential model) is symbolically defined, and then a function NLregression is called to estimate the function parameters:

sym x;                            % symbolic variables
a=sym('a',[3 1]);                 % symbolic parameters
f=@(x,a)a(1)*exp(a(2)*x)+a(3)|    % symbolic function
[a er]=NLregression(data(:,1),data(:,2),f,3);  % call regression

The code below estimates the M parameters of the regression function based on Newton’s method.

function [a er]=NLregression(x,y,f,M)   % nonlinear regression
N=length(x);                  % number of data samples
a=sym('a',[M 1]);             % M symbolic parameters
F=[];
for i=1:N
    F=[F; f(x(i),a)];         % evaluate function at N given data points
end
J=jacobian(F,a);              % symbolic Jacobian
J=matlabFunction(J);          % convert to a true function
F=matlabFunction(F);
a1=ones(M,1);                 % initial guesses for M parameters
n=0;                          % iteration index
da=1;
while da>tol                  % terminate if no progress made
    n=n+1;
    a0=a1;                    % update parameters
    J0=J(a0(1),a0(2));        % evaluate Jacobian based on a0
    Jinv=pinv(J0);            % pseudo inverse of Jacobian
    r=y-F(a0(1),a0(2));       % residual
    a1=a0+Jinv*r;             % Newton iteration
    er=norm(r);               % residual error
    da=norm(a0-a1);           % progress
end
end

where tol is a small value for tollerance, such as 10^{−6}.

Example 1 We first resolve the simple linear regression problem considered previously:

X = [1 2 4], y = [1.6, 1.8, 3.2]^{T}

We have

x̄ = 2.33, ȳ = 2.20, σ_{xy}^{2} = 0.87, σ_{x}^{2} = 1.56

and get

b = \frac{σ_{xy}^{2}}{σ_{x}^{2}} = 0.557, a = ȳ − b x̄ = 0.9

which are the same as the results obtained previously by the linear regrsssion pseudo inverse method.

Consider three sets of data (shown as the solid dots in the three columns in the plots below) are fitted by three different models (shown as the open circles in the plots):

1. Linear model: y = f_{1}(x) = ax + b

2. quadratic model: y = f_{2}(x) = ax^{2} + bx + c

3. Exponential model: y = f_{3}(x) = a e^{bx} + c

These are the parameters for the three models to fit each of the three datasets:

P arameters :  a  b  c  error
Model 1 :  30.337 51.652  7.4294
Dataset 1  Model 2 :  0.076 30.034  51.580 7.4223
Model 3 : 5933.214 0.005 −5881.634 7.4221
Model 1 :  24.519 20.089  21.0362
Dataset 2  Model 2 :  4.459 6.625  15.855 8.4559
Model 3 :  29.791 0.346  −10.202 9.7402
Model 1 :  32.608 22.008  53.0724
Dataset 3  Model 2 :  10.391 −9.089  12.140 28.3211
Model 3 :  2.197 0.885  27.241 8.3393

We see that the first dataset can be fitted by all three models equally well with similar error (when a ≈ 0 in model 2, it is approximately a linear functions), the second dataset can be well fitted by either the second or the third model, while the third dataset can only be fitted by the third exponential model.

Example 2

Based on the following three functions with four parameters a = 1, b = 2, c = 3, d = 4, three sets of 2-D data points are generated (with some random noise added), shown in the figure below as the dots in the three rows.

f_{1}(x) = ax_{1} + bx_{2} − c
f_{2}(x) = exp(ax_{1} + bx_{2}) − cx_{1}x_{2} + d
f_{3}(x) = ax_{1}^{2} − bx_{1}x_{2} + cx_{2}^{2} + d  (5.16)

Then each of three datasets are fitted by the three functions as models while the optimal parameters are found by the Gauss-Newton method.

\begin{aligned}& P arameters : & a & b & c & d & error \\ Dataset 1 & Model 1 : & 0.98 & 1.97 & 3.00 & & 4.13 \\ & Model 2 : & 0.70 & 1.30 & 1.21 & −4.42 & 6.87 \\ & Model 3 : & −0.19 & 0.73 & −0.12 & −3.01 & 18.03 \\ Dataset 2 & Model 1 : & 1.84 & 3.40 & −6.09 & & 14.70 \\ & Model 2 : & 0.99 & 2.01 & 2.97 & 3.98 & 3.75 \\ & Model 3 : & 0.37 & 1.13 & 2.66 & 4.87 & 32.45 \\ Dataset 3 & Model 1 : & −0.19 & 0.30 & −5.34 & & 16.82 \\ & Model 2 : & −0.29 & 0.23 & 2.00 & 4.30 & 13.51 \\ & Model 3 : & 0.94 & 2.05 & 2.99 & 4.02 & 4.09 \end{aligned}

The Levenberg-Marquardt algorithm (LMA) or damped least-squares method is a method widely used to minimize a sum-of-squares error ε as in Eq. (5.3). The LMA is an interpolation or trade-off between the the gradient descent method based on the first order derivative of the error function to be minimized, and the Gauss-Newton method based on both the first and second order derivatives of the error function, both discussed above. As the LMA can take advantage of both methods and is therefore more robust. Even if the initial guess is far from the solution corresponding to the minimum of the objective function, the iteration can still converge toward the solution.

Specifically, the LMA modifies Eq. (5.10)

θ_{n+1} = θ_{n} + J_{f}^{−}(θ_{n})[y − f(X, θ_{n})] = θ_{n} + (J_{n}^{T} J_{n})^{−1}J_{n}^{T} [y − f(X, θ_{n})] (5.17)

to include an extra term proportional to the identity matrix I:

θ_{n+1} = θ_{n} + (J_{n}^{T} J_{n} + λI)^{−1}J_{n}^{T} [y − f(X, θ_{n})]  (5.18)

where λ ≥ 0 is the non-negative damping factor, which is to be adjusted at each iteration. We recognize that this is actaully similar to Eq. (4.46) for ridge regression.

In each iterative step, if λ is small and the error function ε(θ_{n}) decreases quickly, indicating the Gauss-Newton is effective in the sense that the error fucntion is well modeled as a quadratic function by the first and second order terms of the Taylor series, we will keep λ small. On the other hand, if ε decreases slowly, indicating the error function may not be well modeled as a quadratic function, due possibly to the fact that the current θ_{n} is far away from the minimum, then we let λ take a large value, so that the term J_{n}^{T} J_{n} becomes insignificant, and Eq. (5.18) approaches

θ_{n+1} = θ_{n} + λJ_{n}^{T} (y − f(θ)) = θ_{n} − αg_{ε}(θ)  (5.19)

Now θ_{n} is updated along the negative direction of the gradient in Eq. (5.4):

g_{ε}(θ) ∝ J^{T} (f(θ) − y)  (5.20)

with some step size α.

By adjusting the parameter λ, the LMA can switch between either of the two methods for the same purpose of minimizing ε(θ). When the value of λ is small, the iteration is dominated by the Gauss-Newton method that approximate the error function by a quadratic function based on the second order derivatives. If the error reduces slowly, indicating the error function cannot be approximated by a quadratic function, a greater value of λ is used to switch to the gradient descent method.

5.2 Parameter Estimation by Natural Gradient Descent

In Section 4.1, we considered the MLE and MAP methods for estimating the parameter θ of a probability model p(x|θ) by maximizing the likelihood function L(θ|x) or the posterior p(θ|x). In the previous section we also considered how the gradient method is used to estimate the model parameter θ by minimizing the squared error. Obviously we can also use the gradient method in MLE to maximize the likelihood. However, unlike the parameter θ = [θ_{1}, ⋯, θ_{m}]^{T} which is treated as a vector in a Euclidean space spanned by a set of orthonormal basis vectors, the family of distributions p(x|θ) corresponding to all different parameters θ) resides in a distribution space, an m-dimensional Riemannian manifold or space, in which no orthogonal basis is defined and methods based on concepts such as inner product and Euclidean distance are no longer valid.

In this case, the difference between two distributions p(x|θ) and p(x|θ′) in the space can be measured by their KL-divergence KL(p(x|θ)||p_{p}(x|θ + Δθ)), and the gradient method can still be used to find iteratively the optimal parameter that maximize the likelihood L(θ), similar to the steps that lead to Eq. (2.177). Specifically, we desired to find the optimal Δθ that maximally increases the likelihood, which is also subject to the constraint ||Δθ||^{2} < C:

Δθ^{∗} = arg max_{||Δθ||^{2}<C} L(θ + Δθ)  (5.21)

where ||Δθ||^{2} is defined as Eq. (3.253):

||Δθ||^{2} = \frac{1}{2}Δθ^{T} F(θ)Δθ = KL(p(x|θ)||p_{p}(x|θ + Δθ)) < C  (5.22)

To solve this constrained optimization problem we set the derivative of the Lagrangian to zero

\frac{d}{d(Δθ)} [L(θ + Δθ) + μ (\frac{1}{2}Δθ^{T} F(θ)Δθ − C)] = g_{L}(θ) + μF(θ)Δθ = 0

(5.23)

Solving this equation we get the optimal increment:

Δθ^{∗} = −\frac{1}{μ}F^{−1}(θ)g_{L}(θ)  (5.24)

and the optimal parameter can be found iteratively using this method of natural gradient descent (Eq. (3.3.4):

θ_{n+1} = θ_{n} + Δθ_{n} = θ_{n} + λF^{−1}g_{L}(θ_{n}) = θ_{n} + λg̃_{L}(θ_{n})  (5.25)

where λ = 1/μ is the step size, and g̃_{L}(θ_{n}) is the natural gradient:

g̃_{L}(θ_{n}) = F^{−1}g_{L}(θ_{n})  (5.26)

Comparing this iteration with that in Eq. (2.16), we see that the natural gradient method is similar to the Newton-Raphson method in the sense that it is based on not only the gradient g_{J}(θ) for the local first order variation of the likelihood function L(θ) in the distribution space, but also the Fisher information F(θ) for the more global second order curvature in the space, by which the gradient direction is modified so that the iterative search process in the space can be carried out more efficiently.

6 Logistic and Softmax Regression

6.1 Logistic Regression and Binary Classification

All previously discussed regression methods can be considered as supervised binary classifiers, when the regression function f(x, θ) is thresholded by some constant C. Without loss of generality, we will always assume C = 0 in the following. Once the model parameter θ is obtained based on the training set D = {X, y}, every point x in the d-dimensional feature space can be classified into either of two classes C_{−} and C_{+}:

if f(x) = w^{T} x \{ \begin{matrix}< C \\ > C \end{matrix}  then x ∈ \{ \begin{matrix}C_{−} \\ C_{+} \end{matrix}  (6.1)

When the regression function is thresholded by C = 0, it becomes an equation f(x) = 0 representing a hypersurface in the d-dimensional space (a point if d = 1, a curve if d = 2, a surface or hypersurfaceif d ≥ 3), by which the space is partitioned into two regions. In this case, f(x) is called the decision function and the surface decision boundary. Now any point in the space is classified into either of the two classes, depending on whether it is on the positive side of the decision boundary (f(x) > 0) or the negative side (f(x) < 0).

This simple binary classifier suffers from the drawback that the distance between a point x and the decision boundary f(x) = 0 is not taken into consideration, as all points on the same side of the boundary, regardless how far from the boundary, are equally classified into the same class. On the other hand, one would naturally feel more confident about such a classification for a point far away from the boundary, than a point very close to it.

This problem can be addressed by the method of logistic regression, which is similarly trained based on the training set with each sample x_{n} labeled by a binary value y_{n} ∈ {0, 1} for either of the two classes C_{−} and C_{+}. While the logistic regression is based on the linear regression function f(x) = w^{T} x, which is not hard-thresholded by the unit step function u(x) to become either 0 or 1; instead, it is soft-thresholded by a sigmoid function σ(f(x)) = σ(w^{T} x) to become a real value between 0 and 1, interpreted as the probability p(x ∈ C_{+}) for x to belong to C_{+}, and p(x ∈ C_{−}) = 1 − p(x ∈ C_{+}) is the probability for x to belong to C_{−}.

Similar to linear and nonlinear regression methods considered before, the goal of logistic regression is also to find the optimal model parameter θ for the regression function f(x, θ) to fit the training set optimally. But different from linear and nonlinear regression based on the least squares (LS) method that minimizes the squared error in Eq. (5.3), logistic regression is based on maximum likelihood (ML) estimation, which maximizes the likelihood of the model parameter, here θ = w, as discussed below.

The sigmoid function can be either the error function (cumulative Gaussian, erf):

ϕ(x) = ∫_{−∞}^{x} N (u|0, 1) du = \frac{1}{\sqrt{2π}} ∫_{−∞}^{x} exp (−\frac{1}{2N} u^{2}) du  (6.2)

or the logistic function:

σ(x) = \frac{1}{1 + e^{−x}} = \frac{e^{x}}{1 + e^{x}} = \{ \begin{matrix}1 & x → ∞ \\ 0.5 & x = 0 \\ 0 & x → −∞ \end{matrix}  (6.3)

In the following, we will only consider the logistic function for consistency and simplicity. Here is a set of properties of the logistic function:

σ(−x) = \frac{1}{1 + e^{x}} = 1 − \frac{1}{1 + e^{−x}} = 1 − σ(x)

\frac{d}{dx}σ(x) = \frac{e^{−x}}{(1 + e^{−x})^{2}} = σ(x)σ(−x)

x = ln (\frac{σ(x)}{1 − σ(x)}) , e^{x} = \frac{σ(x)}{1 − σ(x)}  (6.4)

Moreover, a parameter w can be included in the logistic function to control its softness when used to threshold the variable x as shown below:

σ(wx) = \{ \begin{matrix}1/2 & w = 0 \\ 1/(1 + e^{−wx}) & w > 0 \\ u(x) & w → ∞ \end{matrix}  (6.5)

We see that the argument x is thresholded by the logistic function with variable softness controlled by parameter w, between the two extreme cases of σ(w x) = 1/2 (no thresholding) when w = 0, and σ(w x) = u(x) when w → ∞ (hard thresholding).

Now the function value f(x) ∈ (−∞, ∞) is mapped to σ(f(x)) ∈ (0, 1), treated as the probability p(x ∈ C_{+}) for x ∈ C_{+}. Specifically, consider the following cases:

• If f(x) = w^{T} x ≫ 0, i.e., x is far away from the plane f(x) = 0 on the positive side, then the probability p(x ∈ C_{+}) = σ(f(x)) is close to 1;

• If f(x) = w^{T} x ≪ 0, i.e., x is far away from the plane f(x) = 0 on the negative side, then the probability p(x ∈ C_{+}) = σ(f(x)) is close to 0, but p(x ∈ C_{−}) = 1 − p(x ∈ C_{+}) for x ∈ C_{−} is close to 1; • If f(x) ≈ 0, i.e., x is close to the plane f(x) = 0, then p(x ∈ C_{+}) = σ(f(x)) ≈

0.5, and p(x ∈ C_{−}) = 1−σ(f(x)) ≈ 0.5, i.e., the probability for x to belong to either C_{+} or C_{−} is about the same.

The value of this logistic function is interpreted as the conditional probability of output ŷ = 1 given data point x and model parameter w:

p(ŷ = 1|x, w) = σ(w^{T} x) = \frac{1}{1 + e^{−w^{T} x}}  (6.6)

and the conditional probability of ŷ = 0 is:

p(ŷ = 0|x, w) = 1−p(y = 1|x, w) = 1−σ(w^{T} x) = σ(−w^{T} x) = \frac{1}{1 + e^{w^{T} x}} (6.7)

Combining these two cases, we get the conditional probability of ŷ = y_{n} (either 1 or 0), as the labeling of a sample x_{n} in the training set:

p(ŷ = y_{n}|x_{n}, w) = p(ŷ = 1|x_{n}, w)^{y_{n}}p(ŷ = 0|x_{n}, w)^{1−y_{n}} = \{ \begin{matrix}p(ŷ = 1|x_{n}, w) & if y_{n} = 1 \\ p(ŷ = 0|x_{n}, w) & if y_{n} = 0 \end{matrix} (6.8)

We further get the likelihood function of w given all N samples in D = {X, y} (assumed i.i.d.):

L(w|D) = p(D|w) = p(y|X, w) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, w)

= ∏_{n=1}^{N} p(ŷ = 1|x_{n})^{y_{n}}p(ŷ = 0|x_{n})^{1−y_{n}} = ∏_{n=1}^{N} σ(w^{T} x_{n})^{y_{n}}(1 − σ(w^{T} x_{n}))^{1−y_{n}} (6.9)

Now the optimal model parameter w can be defined as the one that maximizes the likelihood function L(w|D), or, equivalently, that minimizes the negative log likelihood as the objective function:

J(w) = −l(w|D) = − log L(w|D) = − ∑_{n=1}^{N} [y_{n} log σ(w^{T} x_{n}) + (1 − y_{n}) log(1 − σ(w^{T} x_{n}))]

(6.10)

This is the method of maximum likelihood estimation (MLE), and it produces the most probable estimation of certan parameter of a model based on the given data.

The optimal paramter w can also be found by the method of maximum a posteriori (MAP), which maximizes the posterior probability:

p(w|D) = p(w|X, y) = \frac{p(y|X, w) p(w)}{p(y|X)} ∝ p(y|X, w) p(w)  (6.11)

where p(y|X, w) = L(w|D) is the likelihood found above, p(w) is the prior probability of w without observing the training data D, and the denominator independent of the variable w is dropped as a constant independent of w.

Based on the assumption that all components of w are independent of each other and they have zero mean and the same variance, we let the prior probability be:

p(w) = N (w, 0, σ^{2}I) = \frac{1}{(2πσ^{2})^{N/2}} exp (−\frac{||w||^{2}}{2σ^{2}} ) ∝ exp (−\frac{||w||^{2}}{2σ^{2}} ) (6.12)

where the normalizing factor 1/(2πσ^{2})^{N/2} is independent of w and is therefore dropped. The assumed zero mean is due to the desire that the values of all weights in w should be around zero instead of large values on either positive or negative side, to avoid overfitting, as discussed below.

Now the posterior can be written as

p(w|D) ∝ p(y|X, w) p(w|X) = ∏_{n=1}^{N} p(ŷ_{n} = y_{n}|x_{n}, w) N (0, Σ_{w})

∝ ∏_{n=1}^{N} σ(w^{T} x_{n})^{y_{n}}(1 − σ(w^{T} x_{n}))^{1−y_{n}} exp (−\frac{||w||^{2}}{2σ^{2}} ) (6.13)

We can alternatively find the optimal model parameter w that maximizes this posterior p(w|D) by the mathod of maximum a posteriori (MAP) estimation. To do so, we minimize the negative log posterior as the objective function:

J(w) = − log p(w|D) = − log p(y|X, w) − (−\frac{||w||^{2}}{2σ^{2}} )
= − ∑_{n=1}^{N} y_{n} log σ(w^{T} x) + (1 − y_{n}) log(1 − σ(w^{T} x)) + \frac{||w||^{2}}{2σ^{2}} (6.14)

We can define λ = 1/σ^{2} as a hyperparameter in the second term λ||w||^{2} treated as a penalty term to discourage large values of w, similar to Eq. (4.44) for ridge regression. A larger value of λ will result in smaller values for w, and the logistic function will be more smooth for softer thresholding, and the classification is less affected by possible noisy outliers close to the decision boundary. In other words, by adjusting λ, we can make a proper tradeoff between overfitting and underfitting.

In particular, if σ → ∞ and λ → 0, then exp(−λ||w||^{2}/2) ⇒ 1, i.e., the prior p(w) approaches a uniform distribution (all values of w are equaly likely), then the penalty term can be removed, and Eq. (6.14) becomes the same as Eq. (6.10), i.e., the MAP solution that maximizes the posterior p(w|D) becomes the same as the MLE solution that maximizes the likelihood L(w|D).

Now we can find the optimal w that minimizes the objective function J(w) in either Eq. (6.10) for the MLE method, or Eq. (6.14) for the MAP method. Here we consider the latter case which is more general (with an extra term ||w||^{2}//2σ^{2}). We first find the gradient of objective function:

g_{J}(w) = \frac{d}{dw} J(w) = \frac{d}{dw} (− log p(y|X, w) + \frac{||w||^{2}}{2σ^{2}} )
= − ∑_{n=1}^{N} \frac{d}{dw} [y_{n} log σ(w^{T} x_{n}) + (1 − y_{n}) log(1 − σ(w^{T} x_{n}))] + λw

= − ∑_{n=1}^{N} [\frac{y_{n}}{σ(w^{T} x_{n})} − \frac{1 − y_{n}}{1 − σ(w^{T} x_{n})}] σ(w^{T} x_{n})(1 − σ(w^{T} x_{n})) x_{n} + λw

= ∑_{n=1}^{N} [σ(w^{T} x_{n}) − y_{n}] x_{n} + λw = [x_{1}, ⋯, x_{N}] [ σ(w^{T} x_{1}) − y_{1} ; ⋮ ; σ(w^{T} x_{N}) − y_{N} ] + λw
= Xr + λw  (6.15)

where we have defined a residual vector

r = r(w) = [ r_{1}(w) ; ⋮ ; r_{N}(w) ] = [ σ(w^{T} x_{1}) − y_{1} ; ⋮ ; σ(w^{T} x_{N}) − y_{N} ]  (6.16)

containing all N residuals r_{n} = σ(w^{T} x_{n}) − y_{n} (n = 1, ⋯, N), the difference between the model prediction 0 < σ(w^{T} x_{n}) < 1 and the ground truth labeling y_{n} ∈ {0, 1}. Based on g_{J}(w) we can find the optimal w that minimizes this objective function iteratively by the gradient descent method. We note that the gradient in Eq. (6.15) is based on all N training samples in the training set. If N is large, then the stochastic gradient descent method can be considered based on only one of the N samples in each iteration.

Below is a segment of Matlab code for estimating w by the gradient descent method. Here X contains the N samples x_{1}, ⋯, x_{N} and y contains the corresponding a binary labelings.

The following code segment estimates weight vector w by the gradient descent method:

    X=[ones(1,n); X];          % augmented X with x_0=1
    w=ones(m+1,1);             % initialize weight vector
    g=X*(sgm(w,X)-y)';         % gradient of log posterior
    tol=10^(-6);               % tolerance
    delta=0.1;                 % step size
    while norm(g)>tol          % terminate when gradient is small enough
        w=w-delta*g;           % update weight vector by gradient descent
        g=X*(sgm(w,X)-y)';     % update gradient
    end

where

    function s=sgm(w,X)

        s=1./(1+exp(-w'*X));
    end

Alternatively, this minimization problem for finding optimal w can also be solved by the Newton’s method based on the second order Hessian matrix H_{J}(w) as well as the first order gradient g_{J}(w):

H_{J}(w) = \frac{d}{dw} g_{J}(w) = \frac{d}{dw} (Xr(w) + λ||w||) = X\frac{d}{dw} r(w) + λ

= X\frac{d}{dw} [ r_{1}(w) ; ⋮ ; r_{N}(w) ] + λ = XJ_{r}(w) + λ  (6.17)

where J_{r}(w) = dr(w)/dw is the N × (d + 1) Jacobian matrix of the vector function r(w), whose nth row is

\frac{dr_{n}(w)}{dw} = \frac{d}{dw} [σ(w^{T} x_{n}) − y_{n}] = σ(w^{T} x_{n})[1 − σ(w^{T} x_{n})]x_{n} = z_{n}x_{n} (6.18)

where we have defined z_{n} = σ(w^{T} x_{n})(1 − σ(w^{T} x_{n})), which is always greater than zero as 0 < σ(w^{T} x_{n}) < 1. Now the Jacobian can be written as

J_{r}(w) = [ dr_{1}(w)/dw ; ⋮ ; dr_{N}(w)/dw ] = [ z_{1}x_{1} ; ⋮ ; z_{N}x_{N} ] = [ \begin{matrix}z_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & z_{N} \end{matrix} ] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = ZX^{T}

(6.19)

where Z = diag(z_{1}, ⋯, z_{N}) is a positive definite diagonal matrix (as z_{n} > 0). Substituting J_{r} back into the expression for H_{J}(w) above, we get

H_{J} = XJ_{r} + λ = XZX^{T} + λ  (6.20)

Since Z is positive definite, the following quadratic function is also positive, i.e., for any y, we have

y^{T} H_{J}y = y^{T} XZX^{T} y + λy^{T} y = (X^{T} y)^{T} Z(X^{T} y) + λ||y|| > 0  (6.21)

indicating H_{J} is also positive definite, and the objective function J(w) when approximated by the first three terms of its Taylor series is has a global minimum.

Having found H_{J}(w) as well as g_{l} = J(w) of the objective function J(w), we can solve the minimization problem also by the Newton method:

w_{n+1} = w_{n} − H_{J}(w_{n})^{−1}g_{J}(w_{n}) = w_{n} − (XZX^{T})^{−1}Xr  (6.22)

The following code segment estimates weight vector w by Newton method:

    X=[ones(1,n); X];          % augmented X with x_0=1
    w=ones(m+1,1);             % initialize weight vector
    g=X*(sgm(w,X)-y)';         % gradient of log posterior
    H=X*diag(s.*(1-s))*X';     % Hesian of log posterior
    tol=10^(-6)

    while norm(g)>tol          % terminate when gradient is small enough
        w=w-inv(H)*g;          % update weight vector by Newton-Raphson
        s=sgm(w,X);            % update sigma
        g=X*(s-y)';            % update gradient
        H=X*diag(s.*(1-s))*X'; % update Hessian
    end

Once the model parameter w is available, a given test sample x_{∗} of unknown class can be classified into either of the two classes C_{1} and C_{0}, based on p(y_{∗} = 1|x_{∗}):

if p(y_{∗} = 1|x_{∗}) = σ(w^{T} x_{∗}) \{ \begin{matrix}> 0.5 \\ < 0.5 \end{matrix} , then x_{∗} ∈ \{ \begin{matrix}C_{1} \\ C_{0} \end{matrix}  (6.23)

In summary, we make a comparison between the linear and logistic regression methods both can be used for binary classification. In linear regression, the function value f(x) = w^{T} x is hard-thresholded by the unit step function, so that any data point x is classified to class C_{+} if w^{T} x > 0, or class C_{−} if w^{T} x < 0 in a deterministic fashion (with probability p(x ∈ C_{+}) ∈ {0, 1}. On the other hand, in logistic regression, the linear function f(x) = w^{T} x is soft-thresholded by the logistic function to become p(x ∈ C_{+}) = σ(f(x)) ∈ (0, 1), treated as the probability for x to belong to class C_{+}. Although the same linear model f(x) = w^{T} x is used in both cases, the two methods obtain the model parameter w by minimizing very different objective functions (Eqs. (4.17) and (6.14)).

6.2 Softmax Regression for Multiclass Classification

In a multiclass classification problem, an unlabeled data point x is to be classified into one of K > 2 classes {C_{1}, ⋯, C_{K}}, based on the training set D = {(x_{n}, y_{n}), n = 1, ⋯, N}, where y_{n} ∈ {1, ⋯, K} is an integer indicating x_{n} ∈ C_{y_{n}}

Any binary classifier, such as the logistic regression considered above, can be used to solve such a multiclass classification problem in either of the following two ways:

• One-Vs-One (1V1): construct K(K − 1)/2 binary classifiers for every pair of the K classes based on the training set {(C_{i}, C_{j}) i = 2, ⋯, K, j = 1, ⋯, i−1}, then an unlabeled sample x is classified to one of the K classes with maximum votes out of this many binary classifications.

• One-Vs-Rest (1VR). Convert a K-class problem into K binary problems by regrouping the K classes into two classes C_{+} = C_{i} and C_{−} = {C_{j}|j = 1, ⋯, K, j ≠ i}, and get the corresponding discriminant function f(x) of the binary problem:

if f_{i}(x) \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then \{ \begin{matrix}x ∈ C_{+} = C_{i} \\ x ∈ C_{−} \end{matrix}  (6.24)

which represents quantitatively how much a given x belongs to class C_{+} = C_{i}, instead of C_{−} containing the rest K −1 classes. This process is repeated K times for all i = 1, ⋯, K to get

f_{k}(x) = max{f_{1}(x), ⋯, f_{K}(x)}, (k = 1, ⋯, K)  (6.25)

which indicates x belongs to class C_{k}.

Alternatively, a multiclass problem with K > 2 can also be solved by multinomial logistic or softmax regression, which can be considered as a generalized version of the logistic regression method, based on the softmax function of K variabbles z_{1}, ⋯, z_{K}:

s_{k} = s(z_{k}) = \frac{e^{z_{k}}}{∑_{l=1}^{K} e^{z_{l}}} , (k = 1, ⋯, K)  (6.26)

with the following properties:

0 < s_{k} < 1,  ∑_{k=1}^{K} s_{k} = 1  (6.27)

and

\frac{∂s(z_{i})}{∂z_{j}} = \frac{δ_{ij}e^{z_{i}} ∑_{l=1}^{K} e^{z_{l}} − e^{z_{i}}e^{z_{j}}}{(∑_{l=1}^{K} e^{z_{l}})^{2}}
= δ_{ij}s(z_{i}) − s(z_{i})s(z_{j}) = s(z_{i})[δ_{ij} − s(z_{j})]  (6.28)

where δ_{ij} is the Dirac delta function which is 1 if i = j, but 0 otherwise.

Similar to how the logistic function is used to model the conditional probability p(ŷ = 1|x, w) for a given data point x to belong to class C_{+} based on model parameter w inEq. (6.6), here the softmax function s_{k} defined above is used to model the conditional probability for x to belong to class C_{k} based on some model parameter W = [w_{1}, ⋯, w_{K}]:

p(ŷ = k|x, W) = s_{k} = s(w_{k}^{T} x) = \frac{e^{w_{k}^{T} x}}{∑_{l=1}^{K} e^{w_{l}^{T} x}} , (k = 1, ⋯, K)  (6.29)

where W = [w_{1}, ⋯, w_{K}] is composed of K weight vectors each associated with one of the K classes {C_{1}, ⋯, C_{K}}, to be determined in the training process based on training set D. Same as in logistic regression, here both x and w are augmented d + 1 dimensional vectors.

We note that the inner product w^{T} x = ||w|| ||x|| cos θ of vectors w and x is inversely related to the angle θ between the two vectors, i.e., when compared with other classes, a data sample x has a smaller angular distance to w_{k}, then it has a larger inner product w_{k}^{T} x and thereby greater probability p(ŷ = k|x) to belong to class C_{k}. We therefore see that weight vectors w_{1}, ⋯, w_{K} as the parameters of the softmax regression model actually represent the angular directions of the corresponding classes in the feature space.

Specially, when K = 2, the above becomes logistic functions:

s_{1} = \frac{e^{w_{1}^{T} x}}{e^{w_{0}^{T} x} + e^{w_{1}^{T} x}} = \frac{1}{1 + e^{−(w_{1}−w_{0})^{T} x}} = σ(w^{T} x)
s_{0} = \frac{e^{w_{0}^{T} x}}{e^{w_{0}^{T} x} + e^{w_{1}^{T} x}} = \frac{1}{1 + e^{(w_{1}−w_{0})^{T} x}} = σ(−w^{T} x) = 1 − σ(w^{T} x) (6.30)

where we have defined w = w_{1} − w_{0}, as the vector between the two classes of C_{+} and C_{−}, which is the normal direction of a decision plane separating the two classes. Any point x of unknown class can be therefore classified into either of the two classes depending on whether its projection w^{T} x onto w is positive or negative, i.e., on which side of the decision plane it is located.

For mathamatical convenience, we also label each sample x_{n} in the training set by a binary vector y_{n} = [y_{n1}, ⋯, y_{nK}]^{T}, in addition to its integer labeling y_{n} ∈ {1, ⋯, K}. If y_{n} = k indicating x_{n} ∈ C_{k}, then the kth component of y_{n} is y_{nk} = 1, while other components are zero y_{nl} = 0 for all l ≠ k. Note that all components of y_{n} add up to 1. The N training samples in X = [x_{1}, ⋯, x_{N}] are also labeled by their corresponding binary labelings in Y = [y_{1}, ⋯, y_{N}].

We define the probability for x_{n} ∈ C_{k} as the softmax function based on the linear function w_{k}^{T} x_{n}:

s_{nk} = p(ŷ = k|x_{n}) = \frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} ,  (k = 1, ⋯, K, n = 1, ⋯, N) (6.31)

and the probability for x_{n} to be correctly classified into class C_{y_{n}} can be written as the following product of K factors (of which K − 1 are equal to 1):

p(ŷ = y_{n}|x_{n}, W) = ∏_{k=1}^{K} p(ŷ = k|x_{n}, W)^{y_{nk}} = ∏_{k=1}^{K} s_{nk}^{y_{nk}}  (6.32)

Our goal is to find these weight vectors in W = [w_{1}, ⋯, w_{K}] as the parameter of the softmax model for it to optimally fit the N i.i.d. data points in the training set, so that that the following likelihood is maximized:

L(W|D) ∝ p(D|W) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, W) = ∏_{n=1}^{N} (∏_{k=1}^{K} s_{nk}^{y_{nk}})

= ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} )^{y_{nk}}  (6.33)

Equivalently we can also minimize the negative log likelihood as the objective function:

J(W) = −l(W|D) = − log L(w|D) = − ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} log (\frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} )

= − ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} (w_{k}^{T} x_{n} − log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}w_{k}^{T} x_{n} − ∑_{k=1}^{K} y_{nk} (log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})]

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}w_{k}^{T} x_{n} − log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}]  (6.34)

The last equality is due to the fact that the last summation of K terms is independent of k and ∑_{k=1}^{K} y_{nk} = 1. Also, same as in the cae of K = 2, to avoid overfitting, we could further include in this objective function extra regularization terms λ||w_{k}||^{2}/2, (k = 1, ⋯, K) to penalize large weights for smoother thresholding:

J(W) = −l(W|D) + \frac{λ}{2} ∑_{k=1}^{K} ||w_{k}||^{2}  (6.35)

To find the optimal W that minimize this objective function J(W), we find its gradient vector with respective to each of its K columnns w_{i}, (i = 1, ⋯, K):

g_{i}(W) = \frac{d}{dw_{i}} J(W) = \frac{d}{dw_{i}} [−l(W|D) + \frac{λ}{2} ∑_{k=1}^{K} ||w_{k}||^{2}]

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk} \frac{d}{dw_{i}} (w_{k}^{T} x_{n}) − \frac{d}{dw_{i}} (log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})] + λw_{i}

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}δ_{ik}x_{n} − (\frac{e^{w_{i}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} ) x_{n}] + λw_{i} = ∑_{n=1}^{N} (s_{in} − y_{in}) x_{n} + λw_{i}

= [x_{1}, ⋯, x_{N}] [ s_{i1} − y_{i1} ; ⋮ ; s_{iN} − y_{iN} ] + λw_{i} = [x_{1}, ⋯, x_{N}] [ r_{1} ; ⋮ ; r_{N} ] + λw_{i}
= Xr_{i} + λw_{i}  (6.36)

where r_{n} = s_{in} − y_{in} is the residual, the difference between the model output s_{in} and the binary labeling y_{in}, the ith component of y_{n} that labels x_{n} as well as y_{n}.

As s_{in} is a function of all K weight vectors in W = [w_{1}, ⋯, w_{K}], so is r_{in} = s_{in} − y_{in}, and r_{i} can be explicitly expressed as r_{i}(W). We note that Eq. (6.36) takes the same form as Eq. (6.15) for K = 2. When specially K = 2 with k ∈ {0, 1}, the two equations become the same. We therefore see that logistic regression is actually a special case of softmax regression.

We stack all K such d + 1 dimensional gradient vectors g_{1}, ⋯, g_{K} together, and get a (d + 1)K dimensional gradient vector of the objective function J(W) with respect to all K parameter vectors w_{1}, ⋯, w_{K}:

g_{J} = [ g_{1} ; ⋮ ; g_{K} ]  (6.37)

based on which the optmal parameters in W that minimize the objective function J(W) can be found by the gradient descent method.

We can also find the optimal W by the Newton’s method if we can further find the Hessian matrix H_{J} as the second derivative of J(W), by taking the derivative of the ith gradient vector g_{i}(W) (i = 1, ⋯, K) with respect to the jth weight vector w_{j} (j = 1, ⋯, K) to get the second order derivative of J(W) with respect to both w_{i} and w_{j}, (i, j = 1, ⋯, K):

H_{ij} = \frac{d^{2}}{dw_{j}dw_{i}} J(W) = \frac{d}{dw_{j}} \frac{d}{dw_{i}} J(W) = \frac{d}{dw_{j}} g_{i} = \frac{d}{dw_{j}} [Xr_{i} + λw_{i}]
= X\frac{dr_{i}}{dw_{j}} + δ_{ij}λ = XJ_{ij} + δ_{ij}λ  (6.38)

where J_{ij} = dr_{i}/dw_{j} is the N × (d + 1) Jacobian matrix of r_{i} with respect to w_{j}, of which the nth row is the transpose of the following vector:

\frac{d}{dw_{j}} r_{in} = \frac{d}{dw_{j}} (s_{in} − y_{in}) = \frac{d}{dw_{j}} s_{in} = \frac{d}{dw_{j}} s(w_{i}^{T} x_{n})
= s_{in} (δ_{ij} − s_{jn}) \frac{d}{dw_{j}} (w_{j}^{T} x_{n}) = z_{n}x_{n}  (n = 1, ⋯, N) (6.39)

where we have used Eq. (6.28) and defined z_{n} = s_{in}(δ_{ij} −s_{jn}). Now the Jacobian can be written as

J_{ij} = [ dr_{i1}/dw_{j} ; ⋮ ; dr_{iN}/dw_{j} ] = [ \begin{matrix}z_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & z_{N} \end{matrix} ] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = ZX^{T}  (6.40)

where Z = diag(z_{1}, ⋯, z_{N}) is an N × N diagonal matrix. Substituting J_{r}(W) back into the expression for H_{ij}, we get:

H_{ij} = XJ_{r} + δ_{ij}λ = XZX^{T} + δ_{ij}λ = ∑_{n=1}^{N} s_{in}(δ_{ij} − s_{jn})x_{n}x_{n}^{T} + δ_{ij}λ (6.41)

Here H_{ij} is a matrix of dimension (d + 1) × (d + 1), corresponds to w_{i} and w_{j} (i, j = 1, ⋯, K). We further stack all K × K such matrices H_{ij} together to get the K(d + 1) × K(d + 1) dimensional full Hessian matrix H_{J} of J(W) with respect to all K vectors in W = [w_{1}, ⋯, w_{K}]:

H_{J} = [ \begin{matrix}H_{11} & ⋯ & H_{1K} \\ ⋮ & ... & ⋮ \\ H_{K1} & ⋯ & H_{KK} \end{matrix} ]  (6.42)

Now we can find the optimal parameter W iteratively by the Newton-Raphson method based on both H_{J} and g_{J}:

[ w_{1} ; ⋮ ; w_{K} ]_{n+1} = [ w_{1} ; ⋮ ; w_{K} ]_{n}−H_{n}^{−1} g_{n} = [ w_{1} ; ⋮ ; w_{K} ]_{n}−[ \begin{matrix}H_{11} & ⋯ & H_{1K} \\ ⋮ & ... & ⋮ \\ H_{K1} & ⋯ & H_{KK} \end{matrix} ]_{n}^{−1} [ g_{1} ; ⋮ ; g_{K} ]_{n}

(6.43)

Once W = [w_{1}, ⋯, w_{K}] is available, any unlabeled sample x of unknown class can be classified into one of the K classes with the maximum conditional probability given in Eq. (6.29):

if p(ŷ = k|x) = max_{l} p(ŷ = l|x), then x ∈ C_{k}  (6.44)

Whether we should use softmax regression or K logistic regressions for a problem of K classes C_{1}, ⋯, C_{K} depends on the nature of the classes. The method of softmax regression is suitable if the K classes are mutually exclusive and independent, as assumed by the method. Otherwise, K logistic regression binary classifiers are more suitable.

Below is a Matlab function for estimating the K weight vectors in W = [w_{1}, ⋯, w_{K}] based on the training set D = {X, y and the hyperparameter λ:

function W=softmaxRegression(X,y,lambda)
[d N]=size(X);
K=length(unique(y));        % number of classes
X=[ones(1,N); X];           % augmented data points
d=d+1;
Y=zeros(K,N);
for n=1:N
    Y(y(n),n)=1;            % generate binary labeling Y
end
W=zeros(d*K,1);             % initial guess of K weight vectors
I=eye(K);
s=zeros(K,N);
phi=zeros(K,N);             % softmax functions
gi=zeros(d,1);              % ith gradient
Hij=zeros(d,d);             % ij-th Hebbian
er=9;
tol=10^(-6);
it=0;
while er>tol

it=it+1;
W2=reshape(W,d,K);      % weight vectors in d x K 2-D array
g=[];                   % total gradient vector
for i=1:K
    for n=1:N
        xn=X(:,n);      % get the nth sample
        t=0;
        for k=1:K
            wk=W2(:,k);  % get the kth weight vector
            s(k,n)=exp(wk'*xn);
            t=t+s(k,n);
        end
        phi(i,n)=s(i,n)/t;      % softmax function
    end
    gi=X*(phi(i,:)-Y(i,:))'+lambda*W2(:,i);     % ith gradient
    g=[g; gi];          % stack all gradients into a long gradient vector
end
H=[];                   % total Hessian
z=zeros(N,1);
for i=1:K               % for the ith block row
    Hi=[];
    for j=1:K           % for the jth block in ith row
        for n=1:N
           z(n)=phi(i,n)*((i==j)-phi(j,n));
        end
        Hij=X*diag(z)*X';
        Hi=[Hi Hij];    % append jth block
    end
    H=[H; Hi];          % append ith blow row
end
H=H+lambda*eye(d*K);    % include regulation term
W=W-inv(H)*g;           % update W by Newton's method
er=norm(g);
end
W=reshape(W,d,K);           % reshape weight vector into d x K array
end

Here is the function for the classification of the unlabeled data samples in matrix X based on the weight vectors in W = [w_{1}, ⋯, w_{K}]:

function yhat=softmaxClassify(W,X)
[d N]=size(X);              % dataset to be classified
[d K]=size(W);              % model parameters
X=[ones(1,N); X];           % augmented data points
yhat=zeros(N,1);

for n=1:N                   % for each of the N samples
xn=X(:,n);              % nth sample point
t=0;
for k=1:K
    wk=W(:,k);          % kth weight vector
    s(k)=exp(wk'*xn);
    t=t+s(k);
end
pmax=0;
for k=1:K
    p=s(k)/t;           % probability based on softmax function
    if p>pmax
        kmax=k; pmax=p;
    end
end
yhat(n)=kmax;           % predicted class labeling
end
end

7 Gaussian Process Regression and Classification

7.1 Gaussian Process Regression

The Gaussian process regression (GPR) is yet another regression method that fits a regression function f(x) to the data samples in the given training set. Different from all previously considered algorithms that treat f(x) as a deterministic function with an explicitly specified form, the GPR treats the regression function f(x) as a stochastic process called Gaussian process (GP), i.e., the function value f(x) at any point x is assumed to be a random variable with a Gaussian distribution.

The joint probability distribution of the vector function f(X) = [f(x_{1}), …, f(x_{N})]^{T} for all N i.i.d. samples in the training set X is also Gaussian:

p(f|X) = N (m_{f}, Σ_{f}) = N (0, Σ_{f})  (7.1)

where, for convenience and without loss of generality, we have assumed a zero mean m_{f} = 0. The ultimate goal is to estimate the regression function f(x) in terms of its Gaussian pdf p(f|X).

We also assume y = f(x) + n as the labeling of X in the training set is contaminatd by noise n, with a zero-mean normal pdf

p(n) = N (n, 0, σ_{n}^{2}I)  (7.2)

where the diagonal covariance matrix σ_{n}^{2}I is based on the assumption that all components of n are independent of each other. We further assume noise n is independent of f with Cov(y, n) = 0. Now y = f(X) + n = f + n, as the sum of two zero-mean Gaussian variables, is also a zero mean Gaussian process.

The unknown covariance matrix Σ_{f} can be constructed based on the desired smoothness of the regression function f(x), in the sense that any two function values f(x_{m}) and f(x_{n}) are likely to be more correlated if the distance ||x_{m}−x_{n}|| between the two corresponding points is small, but less so if the distance is large. Such a covariance can be modeled by a squared exponential (SE) kernel function:

Cov(f(x_{m}), f(x_{n})) = exp (−\frac{1}{a^{2}} ||x_{m} − x_{n}||^{2}) = k(x_{m}, x_{n}) = k(x_{n}, x_{m}) (7.3)

where k(x_{m}, x_{n}) is a kernel function of x_{m} and x_{n}. This SE function may take any value between 1 when ||x_{m} − x_{n}|| = 0 and f(x_{m}) = f(x_{n}) are maximally correlated, and 0 when ||x_{m} − x_{n}|| = ∞ and f(x_{m}) and f(x_{n}) are minimally correlated. We therefore see that the SE so constructed mimics the covariance of a smooth function f(x).

The smoothness of the function f(x) plays a very important role as it determines whether the function overfits or underfits the sample points in the training data. If the function is not smooth enough, it may overfit as it may overly affected by some noisy samples; on the other hand, if the function is too smooth, it may underfit as it may not follow the training samples closely enough.

The smoothness of the regression function can be controlled by the hyperparameter α in Eq. (7.3). Consider two extreme cases. If a → ∞, i.e., the value of the SE approaches 1, then f(x_{m}) and f(x_{n}) are highly correlated and f(x) is smooth; on the other hand, if a → 0, i.e., SE approaches 0, then f(x_{m}) and f(x_{n}) are not correlated and f(x) is not smooth. Therefore by adjusting the value of a, a proper tradeoff between overfitting and underfitting can be made.

Given the covariance Cov(f(x_{m}), f(x_{n})) = k(x_{m}, x_{n}) of function values at two different sample points x_{m} and x_{n}, we can represent the covariance matrix of the prior p(f|X) as

Σ_{f} = Cov(f, f) = [ \begin{matrix}k(x_{1}, x_{1}) & ⋯ & k(x_{1}, x_{N}) \\ ⋮ & ... & ⋮ \\ k(x_{N}, x_{1}) & ⋯ & k(x_{N}, x_{N}) \end{matrix} ] = K(X, X) = K (7.4)

Note that the value of all diagonal components are the same k(x_{n}, x_{n}) = 1 (n = 1, ⋯, N); and the greater the distance ||x_{m} − x_{n}||, the farther away the component k(x_{m}, x_{n}) is from the diagonal, and the lower value it takes.

Once the conditional probability p(f|X) = N (0, Σ_{f}) is estimated based on the training set X = [x, ⋯, x_{N}], we can get the mean and covariance of f_{∗} = f(X_{∗}) of a test dataset X_{∗} = [x_{1∗}, ⋯, x_{M∗}] containing M unlabeled test samples. As both f_{∗} and f are the same zero-mean Gaussian process, their joint distribution (given X_{∗} as well as X) is also a zero-mean Gaussian:

p ([ \begin{matrix}f \\ f_{∗} \end{matrix} ] |X, X_{∗}) = N ([ \begin{matrix}f \\ f_{∗} \end{matrix} ] , [ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{f} & Σ_{ff_{∗}} \\ Σ_{f_{∗}f} & Σ_{f_{∗}} \end{matrix} ])
= N ([ \begin{matrix}f \\ f_{∗} \end{matrix} ] , [ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}K & K_{∗} \\ K_{∗}^{T} & K_{∗∗} \end{matrix} ])  (7.5)

where

\{ \begin{matrix}Σ_{ff_{∗}} = Cov(f, f_{∗}) = K(X, X_{∗}) = K_{∗} \\ Σ_{f_{∗}f} = Cov(f_{∗}, f) = K(X_{∗}, X) = K_{∗}^{T} \\ Σ_{f_{∗}} = Cov(f_{∗}, f_{∗}) = K(X_{∗}, X_{∗}) = K_{∗∗} \end{matrix}  (7.6)

can all be constructed based on the SE function, same as how K(X, X) = K is constructed.

Having found the joint pdf p(f, f_{∗}|X, X_{∗}) of both f and f_{∗} in Eq. (7.5), we can further find the conditional distribution p(f_{∗}|f, X, X_{∗}) of f_{∗} given f (as well as X and X_{∗}), based on the properties of the Gaussian distributions discussed in Section 3.1.3:

p(f_{∗}|f, X, X_{∗}) = N (m_{(f_{∗}|f)}, Σ_{(f_{∗}|f)})  (7.7)

with the mean and covariance

\{ \begin{matrix}m_{(f_{∗}|f)} & = & E[f_{∗}|f] = K_{∗}^{T} K^{−1}f \\ Σ_{(f_{∗}|f)} & = & Cov(f_{∗}|f) = K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} \end{matrix}  (7.8)

Similarly, we can also find the joint distribution of f_{∗} and y = f + r, both of which are zero-mean Gaussian:

p(y, f_{∗}|X, X_{∗}) = N ([ \begin{matrix}y \\ f_{∗} \end{matrix} ] , [ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{y} & Σ_{yf_{∗}} \\ Σ_{f_{∗}y} & Σ_{f_{∗}} \end{matrix} ])  (7.9)

where

Σ_{y} = Cov(y, y) = Cov(f + n, f + n) = Cov(f, f) + 2 Cov(f, n) + Cov(n, n)
= Cov(f, f) + Cov(r, r) = Σ_{f} + Σ_{n} = K + σ_{n}^{2}I
Σ_{yf_{∗}} = Cov(y, f_{∗}) = Cov(f + n, f_{∗}) = Cov(f, f_{∗}) + Cov(f, n_{∗})
= Cov(f, f_{∗}) = Σ_{ff_{∗}} = K_{∗}  (7.10)

where Cov(f, n) = Cov(f_{∗}, n) = 0. Now we have

Cov [ \begin{matrix}y \\ f_{∗} \end{matrix} ] = [ \begin{matrix}Σ_{y} & Σ_{yf_{∗}} \\ Σ_{f_{∗}y} & Σ_{f_{∗}} \end{matrix} ] = [ \begin{matrix}K + σ_{n}^{2}I & K_{∗} \\ K_{∗}^{T} & K_{∗∗} \end{matrix} ]  (7.11)

We can now further get the desired conditional distribution of f_{∗} given X_{∗} as well as the training set D = {X, y}:

p(f_{∗}|X_{∗}, D) = p(f_{∗}|X_{∗}, X, y) = N (m_{(f_{∗})}, Σ_{(f_{∗})})  (7.12)

with the mean and covariance

\{ \begin{matrix}m_{(f_{∗})} & = & E[f_{∗}|y] = K_{∗}^{T} (K + σ_{n}^{2}I)^{−1} y \\ Σ_{(f_{∗})} & = & Cov(f_{∗}|y) = K_{∗∗} − K_{∗}^{T} (K + σ_{n}^{2}I)^{−1} K_{∗} \end{matrix}  (7.13)

We see that these expressions are the same as those in Eq. (4.87) previously obtained by Bayesian linear regression, i.e., the same results can be derived by two very different methods.

In the special noise-free case of n = 0 (with zero mean and zero covariance σ_{n}^{2} = 0), we have y = f + n = f, and the mean and covariance in Eq. (7.13) become the same as Eq. (7.8), and Eqs. (7.12) and (7.7) become the same:

p(f_{∗}|X_{∗}, X, y) = p(f_{∗}|X_{∗}, X, f)  (7.14)

The Matlab code below for the Gaussian process regression algorithm, takes the training set D = {X, y}, the test samples in X_{∗}, and the variance σ_{n}^{2} of the noise as the input, and generates as the output the posterior p(f_{∗}|D, X_{∗}), in terms of its mean as the GPR regression function, and its covariance matrix, of which the variances on the diagonal represent the confidence or certainty of the regression.

function [fmean fcov]=GPR(X,y,Xstar,sigmaN)   % given training set D={X,y}, test sample Xstar
                                          % find regression function fmean and covariance
Sy=Kernel(X,X)+sigmaN^2*eye(length(X));   % Covariance of training samples
Syf=Kernel(X,Xstar);                      % Covariance of training and test samples
Sf=Kernel(Xstar,Xstar);                   % Covariance of test samples
fmean=Syf'*inv(Sy)*y;                     % mean of posterior as the regression function
fcov=Sf-Syf'*inv(Sy)*Syf;                 % covariance of posterior
end

The follwoing code is for a function that generates the squared exponential kernel matrix based on two sets of data samples in X1 and X2:

function K=Kernel(X1,X2)
a=1;            % parameter that controls smoothness of f(x)
m=length(X1);
n=length(X2);
K=zeros(m,n);
for i=1:m
    for j=1:n
        d=norm(X1(:,i)-X2(:,j));
        K(i,j)=exp(-(d/a)^2);
    end
end
end

xstar=xmin:0.1:xmax;                % test samples
n=length(xstar);                    % number of test samples
Kss=Kernel(xstar,xstar);            % K(x*,x*), covariance of prior
[L p]=chol(Kss);                    % Cholesky decomposition
for i=1:k
    plot(xstar,L'*randn(n,1))       % plot k sample curves from prior
    hold on
end
hold off
x=xstar(randi(n,1,N)));             % get N random samples from xstar
y=f(xstar)+randn(1,n)*sigma_n^2;    % get their noisy labeling values
Kxx=Kernel(x,x)+sigma_n^2*eye(N);   % K(x,x)
Kxs=Kernel(x,xstar);                % K(x,x*)
Ksx=Kxs';                           % K(x*,x)
fmean=Ksx*inv(Kxx)*y;               % mean of posterior, regression function
fcov=Kss-Ksx*inv(Kxx)*Kxs;          % covariance of posterior
plot(xstar,fmean,xstar,fmean+diag(fcov),xstar,fmean-diag(fcov));

    [L p]=chol(fcov);                   % Cholesky decomposition
    for i=1:k
        plot(xstar,L'*randn(n,1))       % plot k sample curves drawn from posterior
        hold on
    end
    hold off

Example:

A sinusoidal function f(x) = cos(2πfx) (f = 0.2) contaminated by Gaussian noise e with p(r) = N (0, σ_{r}^{2}) (σ_{r} = 0.2) is sampled at N = 7 random positions over the duration of the function (0, 0). These samples form the training dataset D = {(x_{n}, y_{n}) (n = 1, ⋯, N)}.

Then the method of GPR is used to estimate the mean and variance of f_{i∗} = f(x_{i∗}) at 91 test points x_{i∗}, evenly distributed over the duration of the function between the limits x_{min} = 0 and x_{max} = 9,

The 8 panels of the figure below show the estimated mean E[f_{∗}], and the variance ± Var[f_{∗}], both above and below the mean, to form a region of uncertainty of the estimates. The wider the region, the more uncertain the results are (less confidence).

The top left panel represents the prior probability p(f_{∗}|x_{∗}) of the estimated model output f_{∗} = ŷ_{∗} at point x_{∗} with no knowledge based on the training set x (N = 0 training samples), which is assumed to have zero-mean E[f_{∗}] = 0 and unity variance Var[f(x_{∗})] = k(x_{∗}, x_{∗}) = 1. The subsequent seven panels show the mean and variance of the posterior p(f_{∗} = ŷ_{∗}|x_{∗}, D) based on N = 1, ⋯, 7 training samples in the training setD. Note that wherever a training sample x becomes available, the estimated mean μ_{f_{∗}|y} is modified to be close to the true function value f(x), and the variance of the posterior around x is reduced, indicating the uncertainty of the estimate is reduced, or the confidence is much increased. When all N = 7 training samples are used, the general shape of the original function is reasonably reconstructed.

The 8 panels in the figure below show 12 sample curves drawn from each posterior p(f_{∗}|x_{∗}, D). They can be treated as different interpolations for the N observed samples in training set D, and any of them can be used to predict the outputs y_{∗} at any input points x_{∗}. The mean of these samples shown as the red curve is simply the same as E[f_{∗}] shown in the previous figure.

We see that wherever a training sample f(x) is available at x, the variance and thereby the width of the region of uncertainty is reduced, and correspondingly the sample curves in the neighborhood of x are more tightly distributed. Specifically, we see that the sample curves drawn from the prior in the top left panel are very loosely distributed, while those samples drawn from the posterior obtained based on all N = 7 training points in the bottom right panel are relatively tightly distributed. In particular, note that due to lack of training data in the interval 5 < x < 8, the region of uncertainty is significantly wider than else where, and the sample curves are much more loosely located.

The parameter α of the kernel function plays an important role in the GPR results, as shown in the three panels in the figure below, corresponding respectively to a values of 0.5, 1.5, and 3.0. When a is too small (top panel), the model may overfit the data, as the regression function fits the training samples well but it not smooth and it does not interpolate regions between the training samples well (with large variance); one the other hand, when a is too large (bottom panel), the model may underfit the data, as the regression curve is smooth, but it does not fit the training samples well. We therefore see that we need to choose the value of a carefully for a proper tradeoff between overfitting and underfitting (middle panel).

7.2 Gaussian Process Classifier – Binary

In both logistic regression and softmax regression considered previously, we convert the linear regression function ŷ = f(x) = x^{T} w to the probability p(ŷ = k|x, W) for x ∈ C_{k}, by either the logistic function σ(f(x)) for binary classification, or the softmax function s(f(x)) for multiclass classification. Also, in Gaussian process regression (GPR), we treat the regression function f(x) as a Gaussian process. Now we consider the Gaussian process classification (GPC) based on the combination of both logistic/softmax regression and Gaussian process regression. We will consider binary GPC based on the logistic function in this section, and then multiclass GPC based on softmax function in the following section.

Here in binary GPC, we assume the training samples are labeled by 1 or -1 (instead of 0) for mathematical convenience, i.e., x_{n} ∈ C_{+} if y_{n} = 1 or x_{n} ∈ C_{−} if y_{n} = −1. Similar to logistic programming, here we also convert the regression function f(x), now a Gaussian process, into the probability for x ∈ C_{k} by the logistic function σ(f(x)). However, as a function of this random argument, σ(f(x)) is also random. We therefore define p(y = 1|x, D) as the expectation of σ(f(x)) with respect to f(x):

p(y = 1|x, D) = E_{f}[σ(f(x))] = ∫ σ(f(x)) p(f|x, D) df  (7.15)

As function f(x) is marginalized (averaged out) in the integral above, it is hidden instead of explicitly specified, and is therefore called a latent function.

For the training set X = [x_{1}, ⋯, x_{N}], we define f(X) = [f(x_{1}), ⋯, f(x)]^{T}, and express the probability above in vector form:

p(y = 1|D) = E_{f}[σ(f(X))] = ∫ σ(f(X)) p(f|D) df  (7.16)

where p(f|D) is the posterior which can be found in terms of the likelihood p(y|f) = L(f|y) and the prior p(f|X) based on Bayes’ theorem

p(f|D) = p(f|X, y) = \frac{p(y, f|X)}{p(y|X)} = \frac{p(y|f) p(f|X)}{p(y|X)} ∝ p(y|f) p(f|X)  (7.17)

As always, the denominator p(y|X) independent of f is dropped.

We first find the likelihood p(y|f). The Gaussian process f(x) is mapped by the logistic function into the probability for y = 1 for x ∈ C_{+}, or y = −1 for x ∈ C_{−}:

p(y = 1|f(x)) = σ(f(x))
p(y = −1|f(x)) = 1 − p(y = 1|f(x)) = 1 − σ(f(x)) = σ(−f(x)) (7.18)

These two cases can be combined to get the conditional probability of y given f(x):

p(y|f) = σ(y f(x)) = σ(y f) = \frac{1}{1 + e^{−yf}} = \frac{e^{yf}}{1 + e^{yf}}  (7.19)

The likelihood of f for all N i.i.d. samples in the training set X is:

p(y|f) = ∏_{n=1}^{N} p(y_{n}|f_{n}) = ∏_{n=1}^{N} σ(y_{n} f_{n})  (7.20)

We then consider the prior p(f|X), which is assumed to be a zero-mean Gaussian p(f|X) = N (0, Σ_{f}). Here the covariance matrix Σ_{f} can be constructed based on the training set X, the same as in GPR. Specifically, the covariance between f(x_{m}) and f(x_{n}) in the mth row and nth column of Σ_{f} is modeled by the squared exponential (SE) kernel:

cov(f(x_{m}), f(x_{n})) = k(x_{m}, x_{n}) = exp (−\frac{1}{a^{2}} ||x_{m} − x_{n}||^{2}) , (m, n = 1, ⋯, N)

(7.21)

Such a covariance matrix is desired for GPC as well as GPR, so that f(x_{m}) and f(x_{n}) are more correlated if x_{m} and x_{n} are close together, but less so if they are farther apart. However the justification for such a property is different. In GPR, this property is desired for the smoothness of the regression function; while here in GPC, this property is also desired so that function values f(x_{m}) and f(x_{n}) for two samples x_{m} and x_{n} close to each other in the feature space are more correlated and therefore the two samples are more likely to be classified into the same class, but less so if they are far apart.

Also, as discussed in GPR, here the parameter a in the SE controls the smoothness of the function. If a → ∞, the value of the SE approaches 1, then f(x_{m}) and f(x_{n}) are highly correlated and f(x) is very smooth; but if a → 0, SE approaches 0, then f(x_{m}) and f(x_{n}) are not correlated and f(x) is no longer smooth. By adjusting a, a proper tradeoff can be made between overfitting and underfitting.

Having found both the likelihood p(y|f) and the prior p(f), we can get the posterior in Eq. (7.17):

p(f|D) ∝ p(y|f) p(f|X) = p(y|f) N (0, K) = ∏_{n=1}^{N} σ(y_{n} f_{n}) \frac{1}{(2π)^{d/2}|K|^{1/2}} exp (−\frac{1}{2}f^{T} K^{−1}f)

∝ ∏_{n=1}^{N} σ(y_{n} f_{n}) exp (−\frac{1}{2}f^{T} K^{−1}f)  (7.22)

Note that the likelihood p(y|f) is not Gaussian, as the binary labeling y of the training set X is not continuous. Consequently, the posterior p(f|D), as a product of the Gaussian prior and non-Gaussian likelihood, is not Gaussian. However, we can carry out Laplace approximation and still approximate the posterior as a Gaussian:

p(f|D) ≈ N (m_{f|D}, Σ_{f|D})  (7.23)

of which the mean m_{f|D} and covariance Σ_{f|D} are to be obtained in the following. We further get the log posterior denoted by ψ(f):

ψ(f) = log p(f|D) = ∑_{n=1}^{N} log σ(y_{n} f_{n})− \frac{N}{2} log(2π)− \frac{1}{2} log |K|− \frac{1}{2}f^{T} K^{−1}f (7.24)

The two middle terms are constant independent of f and can therefore be dropped.

Now that the posterior p(f|D) is approximated as a Gaussian, we can find the gradient vector g_{ψ}(f) and Hessian matrix H_{ψ}(f) of the log posterior:

g_{ψ}(f) = \frac{d}{df} ψ(f) = −Σ_{f|D}^{−1}(f − m_{f|D})  (7.25)
H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ}(f) = −Σ_{f|D}^{−1}  (7.26)

On the other hand, we can also find g_{ψ}(f) and H_{ψ}(f) by directly taking the first and second order derivatives of the log posterior::

g_{ψ}(f) = \frac{d}{df} ψ(f) = \frac{d}{df} log p(y|f) − \frac{d}{df} (\frac{1}{2}f^{T} K^{−1}f) = w − K^{−1}f (7.27)

H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ}(f) = \frac{d}{df} (w − K^{−1}f) = W − K^{−1}  (7.28)

where we have defined

w = \frac{d}{df} log p(y|f),  W = \frac{d^{2}}{df^{2}} log p(y|f) = \frac{dw}{df}  (7.29)

are respectively the gradient vector and Hessian matrix of the log function

log p(y|f) = log ∏_{n=1}^{N} σ(y_{n} f_{n}) = ∑_{n=1}^{N} log σ(y_{n} f_{n}) = ∑_{n=1}^{N} log (\frac{1}{1 + exp(−y_{n}f_{n})})

(7.30)

We first find the first and second order derivatives with respect to a single component f_{n} of f:

\frac{d}{df_{n}} log p(y_{n}|f_{n}) = \frac{d}{df_{n}} log σ(y_{n}f_{n}) = \frac{d}{df_{n}} log (\frac{1}{1 + exp(−y_{n}f_{n})}) = \frac{y_{n}}{1 + exp(y_{n}f_{n})}
\frac{d^{2}}{df_{n}^{2}} log p(y_{n}|f_{n}) = \frac{d^{2}}{df_{n}^{n}} log σ(y_{n}f_{n}) = \frac{d}{df_{n}} (\frac{y_{n}}{1 + exp(y_{n}f_{n})}) = \frac{− exp(−y_{n}f_{n})}{(1 + exp(−y_{n}f_{n}))^{2}} (7.31)

where we have used the facts that y_{n} = ±1 and y_{n}^{2} = 1. We then find

w = \frac{d}{df} log p(y|f) = [ d/df_{1} ; ⋮ ; d/df_{N} ] ∑_{n=1}^{N} log p(y_{n}|f_{n}) = [ d log p(y_{1}|f_{1})/df_{1} ; ⋮ ; d log p(y_{N}|f_{N})/df_{N} ] = [ ⋮_{\frac{y_{N}}{1+e^{y_{N} f_{N}}}}^{\frac{y_{1}}{1+e^{y_{1}f_{1}}}} ]

(7.32)

and

W = \frac{d^{2}}{df^{2}} log p(y|f) = [ \begin{matrix}\frac{∂^{2}}{∂f_{1}∂f_{1}} & ⋯ & \frac{∂^{2}}{∂f_{1}∂f_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}}{∂f_{N} ∂f_{1}} & ⋯ & \frac{∂^{2}}{∂f_{N} ∂f_{N}} \end{matrix} ] ∑_{n=1}^{N} log p(y_{n}|f_{n})

= diag [\frac{d^{2}}{df_{1}^{2}} log p(y_{1}|f_{1}), ⋯, \frac{d^{2}}{df_{N}^{2}} log p(y_{N}|f_{N})] = diag [\frac{−e^{−y_{1}f_{1}}}{(1 + e^{−y_{1}f_{1}})^{2}} , ⋯, \frac{−e^{−y_{N} f_{N}}}{(1 + e^{−y_{N} f_{N}})^{2}} ] (7.33)

Equatiing the two expressions for H_{ψ}(f) in Eqs. (7.26) and (7.28) we get:

H_{ψ}(f) = W − K^{−1} = −Σ_{f|D}^{−1},  i.e.,  Σ_{f|D} = (K^{−1} − W)^{−1} (7.34)

Equating the two expressions for g_{ψ}(f) in Eqs. (7.25) and (7.27) we get:

g_{ψ}(f) = −Σ_{f|D}^{−1}(f−m_{f|D}) = w−K^{−1}f  i.e.,  m_{f|D} = f+Σ_{f|D} (w − K^{−1}f) = f−H_{ψ}^{−1}g_{ψ}

(7.35)

Substituting Σ_{f|D} given in Eq. (7.34) into the equation and solving it for m_{f|D}, we get

m_{f|D} = f + (K^{−1} + W)^{−1}(w − K^{−1}f)  (7.36)

We recognize this is actaully the iteration of Newton’s method solving equation g_{ψ}(f) = 0, in consistence with the fact that when f = m_{f|D}, the log posterior ψ(f) = log p(f|D), assumed to be Gaussian, reaches maximum with g_{ψ}(f) = 0. However, we note that in the equation above, m_{f|D} as a value of f is expressed in terms of both w and W, which in turn are functions of f as given in Eqs. (7.32) and (7.33), i.e., m_{f|D} given above is not in a closed form, as f and parameters w and W are interdependent. We instead need to carry out an iteration during which both f and the parameters w and W are updated alternatively:

f_{n+1} = f_{n} − H_{ψ}^{−1} g_{ψ} = f_{n} + (K^{−1} − W)^{−1} (w − K^{−1}f_{n})
= f_{n} + (K^{−1} − W)^{−1} [−(K^{−1} − W)f_{n} + w − Wf_{n}]
= (K^{−1} − W)^{−1} (w − Wf_{n})  (7.37)

This iterative process converges to f = m_{f|D}, at which p(f|D) is maximized, and

g_{ψ}(f) = \frac{d}{df} ψ(f) = w − K^{−1}f = 0, i.e., f = m_{f|D} = K w  (7.38)

Now that w and W as well as f = m_{f|D} are available, we can further obtain Σ_{f|D} in Eq. (7.34), and the posterior p(f|D) ≈ N (m_{f|D}, Σ_{f|D}).

Having found p(f|D), we can proceed to carry out classification of any test set X_{∗} based on the probability (similar to Eq. (7.16)):

p(y_{∗} = 1|X_{∗}, D) = E_{f}[σ(f(X_{∗}))] = ∫ σ(f(X_{∗})) p(f|X_{∗}, D) df  (7.39)

We first approximate the posterior of f_{∗} = f(X_{∗}) in the equation as a Gaussian:

p(f|X_{∗}, D) ≈ N (m_{f_{∗}}, Σ_{f_{∗}})  (7.40)

where the mean m_{f_{∗}} and covariance Σ_{f_{∗}} can be obtained based on the fact that both f_{∗} = f(X_{∗}) and f = f(X) are the same Gaussian process, i.e., their joint probability p(f, f_{∗}) is a Gaussian. The method is therefore the same as what is discussed in the method of GPR. Specifically, we take the following steps:

• Find the mean m_{f_{∗}|f} = E(f_{∗}|f) and covariance Σ_{f_{∗}|f} = Cov(f_{∗}|f) of p(f_{∗}|X_{∗}, X, f) conditioned on the latent function f, same as in Eq. (7.8) for GPR:

\{ \begin{matrix}m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f \\ Σ_{f_{∗}|f} = K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} \end{matrix}  (7.41)

• Find m_{f_{∗}} = E_{f}(f_{∗}) as the expectation of m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f conditioned on f, i.e. find the average of m_{f_{∗}|f} over f based on p(f|D) (marginalize over f):

m_{f_{∗}} = E_{f}(m_{f_{∗}|f}) = E_{f} (K_{∗}^{T} K^{−1}f) = K_{∗}^{T} K^{−1}E_{f}(f) = K_{∗}^{T} K^{−1}m_{f}
= K_{∗}^{T} K^{−1}K w_{m_{f}} = K_{∗}^{T} w_{m_{f}}  (7.42)

where m_{f} = E_{f}(f) = Kw_{m_{f}} given in Eq. (7.38).

• Find the covariance Σ_{m_{f∗|f}} = E_{f}[(m_{f_{∗}|f} − m_{f_{∗}})^{2}]. Given the covariance Σ_{f|D} = (K^{−1} −W)^{−1} of f in Eq. (7.34), we can further find the covariance of m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f as a linear combination of f (Recall if y = Ax, then Σ_{y} = AΣ_{x}A^{T}):

Σ_{m_{f∗|f}} = K_{∗}^{T} K^{−1} Σ_{f|D} K^{−1}K_{∗} = K_{∗}^{T} K^{−1}(K^{−1} − W)^{−1}K^{−1}K_{∗} (7.43)

• Find the covariance Σ_{f_{∗}} of f_{∗} as the sum of Σ_{f_{∗}|f} = E[(f_{∗} − m_{f_{∗}|f})(f_{∗} − m_{f_{∗}|f})^{T}] for the variation of f_{∗} with respect to m_{f_{∗}|f}, and Σ_{m_{f∗|f}} = E_{f}[(m_{f_{∗}|f} − m_{f_{∗}})^{2}] for the variation of m_{f_{∗}|f} with respect to m_{f_{∗}}:

Σ_{f_{∗}} = Σ_{f_{∗}|f} + Σ_{m_{f∗|f}}
= (K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗}) + (K_{∗}^{T} K^{−1}(K^{−1} − W)^{−1}K^{−1}K_{∗})
= K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} + K_{∗}^{T} K^{−1}[K − K(K − W^{−1})^{−1}K]K^{−1}K_{∗}
= K_{∗∗} − K_{∗}^{T} (K − W^{−1})^{−1}K_{∗}  (7.44)

Here we have used the identity for (A + B)^{−1} given in Section ??.

Now that the posterior p(f|X_{∗}, D) = N (m_{f_{∗}}, Σ_{f_{∗}}) is approximated as a Gaussian with mean and covariance given in Eqs. (7.42) and (7.44), we can finally carry out Eq. (7.16) to find the probability for the test points in X_{∗} to belong to class C_{1}:

p(y_{∗} = 1|X_{∗}, D) = ∫ σ(f_{∗}) p(f_{∗}|X_{∗}, D) df_{∗}
= E_{f}[σ(f(X_{∗}))] = σ(E_{f}(f(X_{∗}))) = σ(m_{f_{∗}}) (7.45)

Same as in GPR, the certainty or confidence of this classification result is represented by the variances on the diagonal of the covariance Σ_{f_{∗}}.

The Matlab code for the essential parts of this algorithm is listed below. Here X and y are for the training data D = {X, y}, and Xs is an array composed of test vectors. First, the essential segment of the main program listed below takes in the training and test data, generates the covariance matrices K, K_{∗}, and K_{∗∗} represented by K, Ks, and Kss, respectively. The function Kernel is exactly the same as the one used for Gaussian process regression. This code segment then further calls a function findPosteriorMean which finds the mean and covariance of p(f|X, y) based on covariance of training data (K = Cov(X) and y, and computes the mean m_{f_{∗}|y} and covariance Σ_{f_{∗}|y} based on Eqs. (7.42) and (7.44), respectively. The sign function of m_{f_{∗}|y} indicates the classification of the test data points in X_{∗}.

    K=Kernel(X,X);                     % cov(f,f), covariance of prior p(f|X)
    Ks=Kernel(X,Xs);                   % cov(f_*,f)
    Kss=Kernel(Xs,Xs);                 % cov(f_*,f_*)
    [Sigmaf W w]=findPosteriorMean(K,y);
                                       % get mean/covariance of p(f|D), W, w
    meanfD=Ks'*w;                      % mean of p(f_*|X_*,D)
    SigmafD=Kss-Ks'*inv(K-inv(W))*Ks;  % covariance of p(f_*|X_*,D)
    ys=sign(meanfD);                   % binary classification of test data
    p=1./(1+exp(-meanfD));             % p(y_*=1|X_*,D) as logistic function

The function findPosteriorMean listed below uses Newton’s method to find the mean and covariance of the posterior p(f|D) of the latent function f based on the training data, and returns them in meanf and covf, respectively, together with w and W for the gradient and Hessian of the likelihood p(y|f), to be used for computing m_{f_{∗}|D} and Σ_{f_{∗}|D}.

function [meanf covf W w]=findPosteriorMean(K,y)
    % K: covariance of prior of p(f|X)
    % y: labeling of training data X
    % w: gradient vector of log p(y|f)
    % W: Hessian matrix of log p(y|f)
    % meanf: mean of p(f|X,y)
    % covf: covariance of p(f|X,y)

    n=length(y);                   % number of training samples
    f0=zeros(n,1);                 % initial value of latent function
    f=f0;
    er=1;
    while er > 10^(-9)             % Newton's method to get f that maximizes p(f|X,y)
        e=exp(-y.*f);
        w=y.*e./(1+e);             % update w
        W=diag(-e./(1+e).^2);      % update W
        f=inv(inv(K)-W)*(w-W*f0);  % iteration to get f from previous f0
        er=norm(f-f0);             % difference between two consecutive f's
        f0=f;                      % update f
    end
    meanf=f;                       % mean of f
    covf=inv(inv(K)-W);            % coviance of f
end

Example 1 The GPC method is trained by the two classes shown in the figure below, represented by 100 red points and 80 blue points drawn from two Gaussian distributions N (m_{0}, Σ_{0}) and N (m_{1}, Σ_{1}), where

m_{0} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , m_{1} = [ \begin{matrix}−1 \\ 0 \end{matrix} ] , Σ_{0} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , Σ_{1} = [ \begin{matrix}1 & 0.9 \\ 0.9 & 1 \end{matrix} ] (7.46)

The 3-D plot of these two normalized Gaussian distributions is also shown in the figure.

Three classification results are shown in the figure below corresponding to different values used in the kernel function. On the left, the 2-D space is partitioned into red and blue regions corresponding to the two classes based on the sign function of m_{f_{∗}|y}; on the right, the 3-D distribution plots of σ(m_{f_{∗}|y}) representing the estimated probability for x_{∗} to belong to either class (not normalized) is shown, to be compared with the original Gaussian distributions from which the traning samples were drawn.

We see that wherever there is evidence represented by the training samples of either class in red or blue, there are high probabilities for the neighboring points to belong to either C_{1} or C_{0} represented by the positive or negative peaks in the 3-D plots. Data points far away from any evidence will have low probability to belong to either class.

We make the following observations for three different values of the parameter α in SE:

• α = 0.4 (top row), the space is partitioned into three regions, with 20 out of 180 training points misclassified. The estimated distribution is smooth.

• α = 0.2 (middle row), the space is fragmented into several more pieces for the two classes (blue islands inside the red region and vice versa), with 5 out of the 180 training points misclassified. The estimated distribution is jagged.

• α = 0.03 (bottom row), the space is partitioned into still more pieces, with all 180 training points correctly classified. The estimated distribution is spiky.

Although the error rate is lowest when α is small, the classificatioin result is not necessarily the best as it may well be an overfitting of the noisy data. We conclude that by adjusting parameter α, we can make proper tradeoff between error rate and overfitting.

7.3 Gaussian Process Classifier – Multi-Class

The binary GPC considered previously can be generalized to multi-class GPC based on softmax function, similar to the how binary classification based on logistic function is generalized to multi-class classification. First of all, we define the following variables for each class C_{k}, (k = 1, ⋯, K) of the K classes {C_{1}, ⋯, C_{K}}:

• Class labeling:

Binary labeling y_{k} = [y_{1k}, ⋯, y_{Nk}]^{T} indicating whether each of the N samples in the training set X = [x_{1}, ⋯, x_{N}] belongs to C_{k}:

y_{nk} = \{ \begin{matrix}1 & if x_{n} ∈ C_{k} \\ 0 & if x_{n} ∉ C_{k} \end{matrix}  (7.47)

In other words, each training sample x_{n} ∈ C_{k} is labeled by K binary variables y_{nk} = 1 and y_{nl} = 0 for all l = 1, ⋯, K, l ≠ k (same as in softmax regression), as well as an integer labeling y_{n} = k.

• Latent functions:

Latent function f_{k} = [f_{1k}, ⋯, f_{Nk}]^{T}, of which the nth component f_{nk} = f_{k}(x_{n}) (n = 1, ⋯, N) is kth Gaussian process associated with C_{k} evaluated at the nth training sample x_{n} in the training set X = [x_{1}, ⋯, x_{N}].

• Probability modeling:

Probability p_{k} = [p_{1k}, ⋯, p_{Nk}]^{T}, of which the nth component p_{nk} (n = 1, ⋯, N) is the probability for x_{n} ∈ C_{k}, modeled by the solftmax function based on f_{nk} (same as in Eq. (6.31) in softmax regression):

p_{nk} = p(ŷ = k|x_{n}) = \frac{e^{f_{nk}}}{∑_{l=1}^{K} e^{f_{nl}}} , (k = 1, ⋯, K, n = 1, ⋯, N)

(7.48)

Note that if y_{nk} = 1 then necessarily y_{nl} = 0 for all l = 1, ⋯, K but l ≠ k, i.e., the K variables y_{n1}, ⋯, y_{nK} are not independent. given p(y_{nk} = 1), we do not need to consider p(y_{nl} = 0) for any l ≠ k.

The probability for x_{n} to be correctly classified into class C_{y_{n}} can be written as the following product of K factors (of which K − 1 are equal to 1, same as Eq. (6.32) in softmax regression):

p(ŷ = y_{n}|x_{n}, f) = ∏_{k=1}^{K} p(y_{nk} = 1|x_{n}, f)^{y_{nk}} = ∏_{k=1}^{K} p_{nk}^{y_{nk}}  (7.49)

Based on y_{k}, f_{k}, and p_{k} for all k = 1, ⋯, K, we further define the following KN dimensional vectors:

y = [ y_{1} ; ⋮ ; y_{K} ] = [ y_{11} ; ⋮ ; y_{N1} ; ⋮ ; y_{1K} ; ⋮ ; y_{NK} ] , f = [ f_{1} ; ⋮ ; f_{K} ] = [ f_{11} ; ⋮ ; f_{N1} ; ⋮ ; f_{1K} ; ⋮ ; f_{NK} ] , p = [ p_{1} ; ⋮ ; p_{K} ] = [ p_{11} ; ⋮ ; p_{N1} ; ⋮ ; p_{1K} ; ⋮ ; p_{NK} ]

(7.50)

The posterior of f given the training set D can be found based on the Bayesian theorem as (same as in Eq. (7.17) in the binary case):

p(f|D) = p(f|X, y) = \frac{p(y|f) p(f|X)}{p(y|X)} ∝ p(y|f) p(f|X)  (7.51)

We first find the likelihood based on all p_{nk} (same as Eq. (6.33) in softmax regression):

L(f|y) ∝ p(y|f) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, f) = ∏_{n=1}^{N} (∏_{k=1}^{K}(p_{nk})^{y_{nk}}) = ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} )^{y_{nk}}

(7.52)

As each sample x_{n} belongs to only one of the K classes, only one of {y_{n1}, ⋯, y_{nK}} can be 1 while all others are 0, consequently, the product above contains only N probabilities raised to the power of y_{nk} = 1, each for one of the N samples belonging to a certain class, while all other probabilities raised to the power of y_{nk} = 0 become 1 and not considered.

We next assume the prior probability of the latent function f_{k} for each class C_{k} to be a zero-mean Gaussian process p(f_{k}|X) = N (0, Σ_{k}), where the covariance matrix Σ_{k} is constructed based on the squared exponential (SE) for its mn-th component:

cov(f_{k}(x_{m}), f_{k}(x_{n})) = cov(f_{mk}, f_{nk}) = k(x_{m}, x_{n}) = exp (−\frac{1}{a^{2}} ||x_{m} − x_{n}||^{2}) , (m, n = 1, ⋯, N)

(7.53)

Then the prior of f containing all K such latent functions is also a Gaussian p(f|X) = N (0, K), where 0 is a KN-dimensional zero vector and K is a KN×KN block diagonal matrix

K = [ \begin{matrix}K_{1} & 0 & ⋯ & 0 \\ 0 & K_{2} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & K_{K} \end{matrix} ]  (7.54)

with K_{1} = ⋯ = K_{K} as defined in Eq. (7.4) on the diagonal. All off-diagonal blocks are zero as the latent functions of different classes are uncorrelated.

Having found both the likelihood p(y|f) and prioor p(f|X), we can write the posterior as

p(f|D) ∝ p(f|X, y) ∝ p(y|f) p(f|X) = ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} )^{y_{nk}} N (0, K)

(7.55)

We note that the likelihood p(y|f) is not Gaussian, as y_{nk} ∈ {0, 1} is a binary labeling, instead of a continuous variable. As a product of the Gaussian prior and non-Gaussian likelihood, the posterior p(f|D) is not a Gaussian process either. However, for convnience, we still assume it is approximately Gaussian p(f|D) ≈ N (f, m_{f|D}, Σ_{f|D}), of which the mean m_{f|D} and covariance Σ_{f|D} are to be found.

Taking log of the posterior, we get

ψ(f) = log p(f|D) = log p(y|f) + log p(f|X) = ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} (f_{nk} − log ∑_{h=1}^{K} e^{f_{nh}}) + log N (0, K)

= y^{T} f − ∑_{n=1}^{N} log ∑_{h=1}^{K} e^{f_{nh}} − \frac{NK}{2} log(2π) − \frac{1}{2} log |K| − \frac{1}{2}f^{T} K^{−1}f  (7.56)

where we have used the fact that ∑_{k=1}^{K} y_{nk} = 1. The gradient of ψ(f) is

g_{ψ} = \frac{d}{df} ψ(f) = \frac{d}{df} (y^{T} f − ∑_{n=1}^{N} log ∑_{h=1}^{K} e^{f_{nh}} − \frac{NK}{2} log(2π) − \frac{1}{2} log |K| − \frac{1}{2}f^{T} K^{−1}f)
= y − p − K^{−1}f  (7.57)

where the second term p comes from

∑_{n=1}^{N} \frac{d}{df} log ∑_{h=1}^{K} e^{f_{nh}} = ∑_{n=1}^{N} [ 0 ; ⋮ ; 0 ; \frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} ; 0 ; ⋮ ; 0 ] = ∑_{n=1}^{N} [ 0 ; ⋮ ; 0 ; p_{nk} ; 0 ; ⋮ ; 0 ] = p  (7.58)

Note that p = p(f) is a function of f.

As the posterior p(f|D) ≈ N (m_{f|D}, Σ_{f|D}) is approximated as a Gaussian, which reaches maximum at its mean m_{f|D}, and so does the log prior ψ(f) = log p(f|D), and the gradient of psi(f) is zero at f = m_{f|D}:

g_{ψ}(f)|_{f=m_{f|D}} = y − p − K^{−1}f|_{f=m_{f|D}} = y − p_{m_{f}} − K^{−1}m_{f|D} = 0, (7.59)

i.e.,

m_{f|D} = K(y − p_{m_{f}})  (7.60)

where p_{m_{f}} = p(m_{f|D}) is the vector defined above evaluated at f = m_{f|D}. We further get the Hessian matrix of ψ(f):

H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ} = \frac{d}{df} (y − K^{−1}f − p) = −K^{−1} − W = −Σ_{f|D}^{−1}

(7.61)

The last equality is due to the property of the Gaussian distribution given in Section 3.1.3, from which we get the KN by KN covariance matrix of p(f|D):

Σ_{f|D} = −H_{ψ}^{−1}(f) = (K^{−1} + W)^{−1}  (7.62)

Here W = dp/df is a KN × KN Jacobian matrix of p, of which the ijkl-th component (i, j = 1, ⋯, N, k, l = 1, ⋯, K) is:

\frac{∂}{∂f_{jl}} p_{ik} = \frac{∂}{∂f_{j}^{l}} [\frac{e^{f_{ik}}}{∑_{h=1}^{K} e^{f_{ih}}} ] = \frac{e^{f_{ik}} (∑_{h=1}^{K} e^{f_{ih}}) δ_{kl} − e^{f_{ik}}e^{f_{jl}}}{(∑_{h=1}^{K} e^{f_{ih}})^{2}} δ_{ij}

= [\frac{e^{f_{ik}}}{∑_{h=1}^{K} e^{f_{ih}}} δ_{kl} − \frac{e^{f_{ik}}e^{f_{jl}}}{(∑_{h=1}^{K} e^{f_{ih}})^{2}} ] δ_{ij} = (p_{ik}δ_{kl} − p_{ik}p_{jl})δ_{ij},(7.63)

and W can be written also in two terms:

W = diag(p) − PP^{T}  (7.64)

where the first term diag(p) is a diagnal matrix containing all KN components of p along the diagnal (k = l and i = j), and P in the second term is a KN by N matrix composed of K N × N diagonal matrices diag(p_{k}):

P = [ diag(p_{1}) ; ⋮ ; diag(p_{K}) ] , diag(p_{k}) = [ \begin{matrix}p_{1k} & 0 & ⋯ & 0 \\ 0 & p_{2k} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & p_{Nk} \end{matrix} ] , (k = 1, ⋯, K)

(7.65) so that

PP^{T} = [ diag(p_{1}) ; ⋮ ; diag(p_{K}) ] [diag(p_{1}), ⋯, diag(p_{K})] = [ \begin{matrix}diag(p_{1}^{2}) & ⋯ & diag(p_{1})diag(p_{K}) \\ ⋮ & ... & ⋮ \\ diag(p_{K})diag(p_{1}) & ⋯ & diag(p_{K}^{2}) \end{matrix} ]

(7.66)

with

diag(p_{k})diag(p_{l}) = [ \begin{matrix}p_{1}^{k}p_{1}^{l} & 0 & ⋯ & 0 \\ 0 & p_{1}^{k}p_{1}^{l} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & p_{Nk}p_{Nl} \end{matrix} ]  (7.67)

Having found both the gradient g_{ψ} = y−p−K^{−1}f and Hessian H_{ψ} = −(K^{−1}+ W), we can further find the mean m_{f|D} at which ψ(f) achieves maximum by the following iteration of Newton’s method:

f_{n+1} = f_{n} − H_{ψ}^{−1}g_{ψ} = f_{n} + (K^{−1} + W)^{−1}(y − K^{−1}f_{n} − p)
= f_{n} + (K^{−1} + W)^{−1}(−(K^{−1} + W)f_{n} + Wf_{n} + y − p)
= (K^{−1} + W)^{−1}(Wf_{n} + y − p)  (7.68)

We note that during the iteration, we need to update not only f, but also p as a functions of f given in Eq. (7.58).

Now that we have got both m_{f|D} and Σ_{f|D} of f, we can further get m_{f_{∗}} and Σ_{f_{∗}} of f_{∗}:

• Get mean m_{f_{∗}}:

Similar to Eq. (7.42) in the binary case, we have the following based on f = K(y − p) in Eq. (7.60):

m_{f_{∗}} = E_{f}(m_{f_{∗}|f}) = E_{f}(K_{∗}^{T} K^{−1}f) = K_{∗}^{T} K^{−1}E_{f}(f)
= K_{∗}^{T} K^{−1} [K(y − p_{m_{f}})] = K_{∗}^{T} (y − p_{m_{f}})  (7.69)

As the K classes are uncorrelated, i.e., K is block-diagonal, the above can be separated into K equations each for one of the classes:

m_{f_{∗}^{k}} = (K_{∗}^{k})^{T} (y^{k} − p^{k}), (k = 1, ⋯, K)  (7.70)

• Get covariance Σ_{f_{∗}}:

Similar to Eq. (7.44) in the binary case, we have the following based on Σ_{f|D} = (K^{−1} + W)^{−1} in Eq. (7.62):

Σ_{m_{f∗|f}} = K_{∗}^{T} K^{−1} Σ_{f|D} K^{−1}K_{∗} = K_{∗}^{T} K^{−1}(K^{−1} + W)^{−1}K^{−1}K_{∗} (7.71)

where W = diag(p)−PP^{T} is given in Eq. (7.64) with p evaluated at m_{f|D}. Then, similar to Eq. (7.44), we get

Σ_{f_{∗}} = K_{∗∗} − K_{∗}^{T} (K + W^{−1})^{−1}K_{∗}  (7.72)

Now we can further get the probability for x_{∗} to belong to C_{k} based on the softmax function in Eq. (7.48)

p_{∗}^{k} = \frac{e^{m_{f∗}^{k}}}{∑_{l=1}^{K} e^{m_{f∗}^{l}}} , (k = 1, ⋯, K)  (7.73)

and classifiy x_{∗} to class C_{k} if p_{∗}^{k} = max{p_{∗}^{1}, ⋯, p_{∗}^{K}}.

Moreover, the certainty or confidence of this classification result can be found from Σ_{f_{∗}}.

The Matlab code for the essential parts of the algorithm is listed below. First, the following code segment carries out the classification of n given test samples based on the N training set D = {X, y}.

K=Kernel(X,X);                        % covariance of prior of p(f|X)
Ks=Kernel(X,Xs);
Kss=Kernel(Xs,Xs);
[meanf Sigmaf p W]=findPosteriorMeanMC(K,y,C);
                                      % find mean and covariance of p(f|D), and W, p
Sigmafy=Kss-Ks'*inv(K+inv(W))*Ks;     % covariance of p(f|D)
p=reshape(p,N,C);
y=reshape(y,N,C);
for k=1:C
    meanfD(:,k)=Ks'*(y(:,k)-p(:,k);   % mean of p(f_*|X_*,D) for kth class
end
for i=1:n                             % for each of n test samples
    d=sum(exp(meanfD(i,:)));          % denominator of softmax function
    [pmax k]=max(exp(meanfD(i,:))/d); % find class with max probability
    ys(i)=k;                          % label ith sample as member of kth class
    pr(i,k)=pmax;                     % probility of ith sample belonging to kth class
end

The code segment above calls the following function which computes the mean and covariance of p(f|X, y) by Newton’s method:

function [meanf Sigmaf p W]=findPosteriorMeanMC(K,y,C)
% find mean and covariance of p(f|X,y) by Newton's method, based on D={X,y}
n=length(y);                    % number of training samples
f0=zeros(n,1);                  % initial value of latent function
f=zeros(n,1);
er=1;
k=0;
while er > 10^(-9)              % find f that maximizes p(f|X,y)
    k=k+1;
    [p W]=findp(f,N,C);         % call function findp to find vector p and matrix W
    f=inv(inv(K)+W)*(W*f0+y-p); % iteratively update value of f
    er=norm(f-f0);              % difference between consecutive iterations

    f0=f;                       % update f
end
meanf=f;
Sigmaf=inv(inv(K)+W);
end

The following function called by the previous function finds vector p and matrix W = diag(p) − PP^{T}:

function [p W]=findp(f,N,C)         % find vector p and matrix W=diag(p)-P*P'
F=reshape(f,N,C)';              % kth row contains N samples of class k
p=zeros(C,N);                   % initialize p
for n=1:N                       % for each of N training samples
    d=sum(exp(F(:,n)));         % sum of all C terms in denominator
    for k=1:C                   % for all C classes
        p(k,n)=exp(F(k,n))/d;
    end
end
P=[];
for k=1:C                       % generate P
    P=[P; diag(p(k,:))];        % stack C diagonal matrices
end
p=reshape(p',N*C,1);            % convert p into a column vector
W=diag(p)-P*P';                 % generate W matrix
end

Example 1: This example shows the classification result of the same dataset of three classes

used before. The top two panels show the distributions of the three classes in the training data set. The bottom two panels show the classification results in terms of the partitioning of the feature space (bottom left) and the posterior distribution p(f_{∗}|X, y, X_{∗}) (bottom right), which can be compared with the distribution of the training set (top right). The confusion matrix of the classification result is shown below, with the error rate 30/600 = 0.045:

[ \begin{matrix}197 & 3 & 0 \\ 0 & 195 & 5 \\ 1 & 21 & 178 \end{matrix} ]  (7.74)

Example 2: This example shows the classification of the XOR data set. The confusion

matrix of the classification result is shown below, with the error rate 13/400 = 0.0325:

[ \begin{matrix}193 & 7 \\ 6 & 194 \end{matrix} ]  (7.75)

We see that in both examples, the error rates of the GPC method are lower than those of the naive Bayesian method. However, the naive Bayesian method does not have the overfitting problem, while in the method of GPC, we may need to carefully adjust the parameter of squared exponential for the kernel functions to make proper tradeoff between overfitting and error rate.

Part III Feature Selection 8 Feature Selection

Feature Selection

8.1 Dimensionality Reduction

The main goal of pattern classification is to classify a set of patterns, objects of interest each described by d features represented as a d-dimensional vector x = [x_{1}, ⋯, x_{d}]^{T} in the feature space, into one of some K classes {C_{1}, ⋯, C_{K}}. In supervised classification, we are to partition the feature space into K regions each corresponding to one of the classes based on the provided training set D = {X, y}, of which each training sample in X = [x_{1}, ⋯, x_{N}] is labeled by the corresponding component in y = [y_{1}, ⋯, y_{N}]^{T}. For example, if y_{n} = {1, 2, ⋯, K}, then x_{n} ∈ C_{y_{n}}. However, different types of labeling can be used depending on the specific classification algorithms, such as binary labeling used in certain regression algorithms considered previously.

When the dimensionality d of the feature space is high, there is the need to reduce it significantly to d′ ≪ d while still maintaining most of the information relevant to the task of classification, i.e., the separability of the classes. This is called feature selection, which can be done by either selecting d′ features directly from the d original ones (there are C_{d}^{d′} = d!/d′!(d − d′)! ways to do so), or generating d′ new features as certain linear combinations of the d original ones. In either case, the separability of the data are to be maximally maintained in the resulting space spanned by the d′ features, so that the classification can be carried out both efficiently and effectively in the much lower dimensional feature space.

8.2 Distances and Separability Measurements

We first consider various distances widely used for feature selection in pattern classification.

• Distance between two data points The distance between two points x and y in the n-dimensional feature space can be measured by the p-norm of the difference x − y:

d_{p}(x, y) = ||x − y||_{p} = (∑_{i=1}^{n} |x_{i} − y_{i}|^{p})^{1/p} 1 ≤ p ≤ ∞  (8.1)

Specially, consider the following three cases:

– p = 1, city block or Manhattan distance:

d_{1}(x, y) = ||x − y||_{1} = ∑_{i=1}^{n} |x_{i} − y_{i}|  (8.2)

– p = 2, Euclidean distance:

d_{2}(x, y) = ||x − y||_{2} = \sqrt{∑_{i=1}^{n} |x_{i} − y_{i}|^{2}}  (8.3)

– p = ∞, the Chebychev distance:

d_{∞}(x, y) = ||x − y||_{∞} = max{|x_{1}|, ⋯, |x_{n}|}  (8.4)

• Intra-class distance

An intra-class distance represents how widely (or narrowly) all samples in each class C_{k} are distributed in the space. It needs to be small for good separability.

– The max diameter:

d_{max}(C_{k}) = max_{x,y∈C_{k}} d(x, y)  (8.5)

– The average diameter:

d_{average}(C_{k}) = \frac{1}{N_{k}(N_{k} − 1)} ∑_{x,y∈C_{k},x≠y} d(x, y)  (8.6)

– The average distance from each sample to the mean vector of the cluster:

\frac{1}{N_{k}} ∑_{x∈C_{k}} d(x, m_{k}), where m_{k} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x  (8.7)

– The covariance of all samples in the class represents the tightness of the samples in the class:

Σ_{k} = \frac{1}{N_{k}} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T}  (8.8)

and its determinant or trace can be used as such a scalar measurement of the tightness:

det Σ_{k} = ∏_{i=1}^{n} λ_{i}, trΣ_{k} = ∑_{i=1}^{n} λ_{i}  (8.9)

As Σ_{k} is positive semi-definite, all of its eigenvalues λ_{i} are nonnegative and so are its determinant and trace.

• Distance between a point and a cluster/class of points – The max and min-distances:

d_{max}(x, C_{k}) = max_{y∈C_{k}} d(x, y), d_{min}(x, C_{k}) = min_{y∈C_{k}} d(x, y) (8.10)

– The centroid distance:

d_{centroid}(x, C_{k}) = d(x, m_{k})  (8.11)

– The Mahalanobis distance

d_{M}(x, C_{k}) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})  (8.12)

• Inter-class distance

An intra-class distance measures the difference between two classes, it needs to be large for good separability.

– The max- and min-distances:

d_{max}(C_{i}, C_{j}) = max_{x∈C_{i},y∈C_{j}} d(x, y), d_{min}(C_{i}, C_{j}) = min_{x∈C_{i},y∈C_{j}} d(x, y)

(8.13)

– The average distance between two clusters is the average of all pair-wise distances (e.g., Euclidean) between members of the two classes:

d_{average}(C_{i}, C_{j}) = \frac{1}{N_{i}N_{j}} ∑_{x∈C_{i},y∈C_{j}} d(x, y)  (8.14)

– The centroid distance is the distance (e.g., Euclidean) between the centroids (mean vectors) of the two classes:

d_{centroid}(C_{i}, C_{j}) = d(m_{i}, m_{j})  (8.15)

– The Bhattacharyya distance:

d_{B}(C_{i}, C_{j}) = \frac{1}{4}(m_{i} − m_{j})^{T} (\frac{Σ_{i} + Σ_{j}}{2} )^{−1} (m_{i} − m_{j})
+ log [\frac{|(Σ_{i} + Σ_{j})/2|}{(|Σ_{i}| |Σ_{j}|)^{1/2}} ]  (8.16)

The first term reflects mostly the difference between the two mean vectors of the two clusters, while the second term reflects the difference between the distributions of the two clusters, which is always positive due to the AM-GM inequality (algebraic mean vs geometric mean) of the arithmetic and geometric means:

\frac{1}{n} ∑_{i=1}^{n} x_{i} ≥ (∏_{i=1}^{n} x_{i})^{1/n}  (8.17)

Note that even when the first term is zero due to m_{i} = m_{j}, the Bhattacharyya distance may still not zero due to the non-zero second term if Σ_{i} ≠ Σ_{j}.

We also define a set of scatter matrices all related to the separability of the classes/clusters in the feature space, based on which different subsets of features can be compared and the best subset can be selected. We will also consider methods for feature selection or feature extraction to reduce the dimensionality of the feature space. Here we assume there are K classes (or clusters) in the dataset containing N data points, and the kth class contains N_{k} data points, i.e., N_{1} + ⋯ + N_{K} = N.

• Total scatter matrix:

Thee total scatter matrix is the same as the covariance matrix:

S_{T} = \frac{1}{N} ∑_{all x}(x − m)(x − m)^{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m)(x − m)^{T}(8.18)

Note that S_{T} rank(S_{T}) ≤ N − 1, due to the mean m = ∑all x x as a constraint.

• Within-class (intra-class) scatter matrix:

S_{W} = ∑_{k=1}^{K} P_{k}Σ_{k} = ∑_{k=1}^{K} \frac{N_{k}}{N} Σ_{k} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T}(8.19)

Note that rank(S_{W}) = ∑_{k=1}^{K}(N_{k} − 1) ≤ N − K, due to the K means m_{k} = ∑_{x∈C_{k}} x/N_{k}, (k = 1, ⋯, K) as K constraints.

• Between-class (inter-class) scatter matrix:

S_{B} = ∑_{k=1}^{K} P_{k}(m_{k} − m)(m_{k} − m)^{T} = ∑_{k=1}^{K} \frac{N_{k}}{N} (m_{k} − m)(m_{k} − m)^{T} (8.20)

Note that rank(S_{B}) ≤ K − 1, due to total mean m = ∑_{k=1}^{K} N_{k}m_{k}/N as a constraint.

We can show that S_{T} = S_{W} + S_{B}, i.e., the total scatter matrix is the sum of within-class and between-class scatter matrices:

S_{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m)(x − m)^{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k} + m_{k} − m)(x − m_{k} + m_{k} − m)^{T}

= \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}[(x − m_{k})(x − m_{k})^{T} + (x − m_{k})(m_{k} − m)^{T} + (m_{k} − m)(x − m_{k})^{T} + (m_{k} − m)(m_{k} − m)^{T}]

= \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T} + \frac{1}{K} ∑_{k=1}^{K} ∑_{x∈C_{k}}(m_{k} − m)(m_{k} − m)^{T}

= ∑_{k=1}^{K} \frac{N_{k}}{N} Σ_{k} + ∑_{k=1}^{K} \frac{N_{k}}{N} (m_{k} − m)(m_{k} − m)^{T} = S_{W} + S_{B}  (8.21)

Obviously, for better separability, we want the within-class scatteredness to be small but the between-class scatteredness to be large. However, we cannot use the scatter matrices directly as they cannot be compared in terms of their sizes. Instead, we use their traces (or determinants) as the scalar measurements for the separability:

J_{B} = tr S_{B} = tr [∑_{k=1}^{C} P_{k}(m_{k} − m)(m_{k} − m)^{T} ]

= ∑_{i=1}^{C} P_{k} tr [(m_{k} − m)(m_{k} − m)^{T} ] = ∑_{i=1}^{C} P_{k} ||m_{k} − m||^{2} (8.22)

and

J_{W} = tr S_{W} = tr [∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T} ]

= ∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}} tr [(x − m_{k})(x − m_{k})^{T} ] = ∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}} ||x − m_{k}||^{2}(8.23)

We see that J_{B} is the weighted sum of the Euclidean distances between m_{k} and m for all C classes, and J_{W} is the weighted sum of the sums of Euclidean distances between all samples x ∈ C_{k} to the mean m_{k} for all K classes C_{1}, ⋯, C_{K}.

To achieve the best separability, we maximize trS_{B} while at the same time minimize trS_{W}. In the same space with a constant S_{T}, maximizing trS_{B} is equivalent to minimizing trS_{W}, due to the relationship S_{B} + S_{W} = S_{T}. To measure the separability across different spaces, S_{B} can still be used but now it needs to be normalized by either S_{T} or equivalently S_{W}:

J_{B/W} = tr(S_{W}^{−1}S_{B}) = tr(S_{B}S_{W}^{−1}), J_{B/T} = tr(S_{T}^{−1}S_{B}) = tr(S_{B}S_{T}^{−1})

(8.24)

The separability measured by the scatter matrices can be used as the criterion for selecting m < n of the n original features to reduce the computational cost while still keeping the effectiveness of the classification. Moreover, we can also generate m new features as the linear combination of the n original ones by a linear transform:

y_{m×1} = (A_{n×m})^{T} x_{n×1}  (8.25)

We desire to find the n × m transform matrix A = [a_{1}, ⋯, a_{m}] so that the separability in the original n-D space is maximally conserved in the new m-D space of lower dimensionality.

• Optimal transformation for maximizing tr(S_{B})

The separability criterion tr (S_{B}^{(y)}) in new m-D space can be expressed as:

J^{(y)}(A) = tr (S_{B}^{(y)}) = tr (A^{T} S_{B}^{x} A) = tr [ a_{1}^{T} ; ⋮ ; a_{m}^{T} ] S_{B}^{x}[a_{1}, ⋯, a_{m}]

= tr [ a_{1}^{T} ; ⋮ ; a_{m}^{T} ] [S_{B}^{x}a_{1}, ⋯, S_{B}^{x}a_{m}] = ∑_{i=1}^{m} a_{i}^{T} S_{B}^{x}a_{i}  (8.26)

To find the optimal transform matrix A that maximizes tr(S_{B}^{(y)}), we consider the following constrained maximization problem:

\{ \begin{matrix}maximize & J(A) = tr(A^{T} S_{B}^{x}A) = ∑_{i=1}^{m} a_{i}^{T} S_{B}^{x}a_{i} \\ subject to & ||a_{j}||^{2} = a_{j}^{T} a_{j} = 1 (j = 1, ⋯, m) \end{matrix}  (8.27)

Here the constraint is to guarantee that the column vectors of A are normalized. Same as the PCA problem discussed in Section 3.5, this constrained optimization problem can be solved by Lagrange multiplier method:

\frac{∂}{∂a_{i}} [J(A) − ∑_{j=1}^{m} λ_{j}(a_{j}^{T} a_{j} − 1)] = \frac{∂}{∂a_{i}} [∑_{j=1}^{m}(a_{j}^{T} S_{B}^{x}a_{j} − λ_{j}a_{j}^{T} a_{j} + λ_{j})]

= \frac{∂}{∂a_{i}} [a_{i}^{T} S_{B}^{x}a_{i} − λ_{i}a_{i}^{T} a_{i}] = 2S_{B}^{x}a_{i} − 2λ_{i}a_{i} = 0, i.e. S_{B}a_{i} = λ_{i}a_{i} (i = 1, ⋯, n) (8.28)

We see that the optimal feature selection transform is the PCA transform which compacts most of the energy/information (representing separability in this context) into m components. The column vectors of A must be the orthogonal eigenvectors of the symmetric matrix S_{B}^{x} corresponding to the m greatest eigenvalues, and the m new features can be obtained by

y_{m×1} = A_{m×n}^{T}x_{n×1} = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{m}^{T} ]_{m×n} x_{n×1}, or y_{i} = ϕ_{i}^{T} x, (i = 1, ⋯, m)

(8.29)

and

J(A) = J(Φ) = ∑_{i=1}^{m} ϕ_{i}^{T} S_{B}ϕ_{i} = ∑_{i=1}^{m} λ_{i}  (8.30)

where the eigenvalues of S_{B} are sorted in descending order λ_{1} ≥ ⋯ ≥ λ_{m} ≥ ⋯ ≥ λ_{n}.

• Optimal transformation for maximizing tr(S_{B/T})

The previous method only maximizes S_{B} without taking into consideration S_{W} or, equivalently, S_{T} = S_{B} + S_{W}. If in the m-D space after the transform S_{W}^{(y)} has also changed as well as S_{B}^{(y)}, then we need to maximize S_{B} while at the same time also minimize S_{W}, in order to maximize the separability. Or, equivalently, we need to maximize S_{B}^{(y)} normalized by the total scatteredness S_{T}^{(y)}. We can therefore use the trace of S_{T}^{−1}S_{B} (or S_{W}^{−1}S_{B}) as the measurement of the separability in the new m-D space:

S_{B/T}^{(y)} = (S_{T}^{(y)})^{−1} S_{B}^{(y)} = (A^{T} S_{T}^{x} A)^{−1}(A^{T} S_{B}^{x}A)
= A^{−1}(S_{T}^{x})^{−1}(A^{T})^{−1}A^{T} S_{B}^{x}A = A^{−1}(S_{T}^{x})^{−1}S_{B}^{x}A = A^{−1}S_{B/T}^{x} A (8.31)

We need to find A = [a_{1}, ⋯, a_{m}] that maximizes the trace of the matrix above in the m-D space of y = A^{T} x. As S_{B/T}^{(y)} (a product of two symmetric matrices) is not a symmetric matrix, the PCA method above cannot be used, we will find the optimal matrix A in some other way.

We first let m = 1, so that A = a is an n × 1 vector that maximizes the following objective function in the 1-D space y = a^{T} x:

(S_{T}^{(y)})^{−1} S_{B}^{(y)} = (a^{T} S_{T}^{x} a)^{−1}(a^{T} S_{B}^{x}a) = \frac{s_{B}^{(y)}}{s_{T}^{(y)}} = \frac{a^{T} S_{B}^{x}a}{a^{T} S_{T}^{x} a} = R(a) (8.32)

This function R(a) is the Rayleigh quotient (Section A.11) of the two symmetric matrices S_{B}^{x} and S_{T}^{x}. The optimal transform vector a that maximizes this R(a) can be found by solving the corresponding generalized eigenvalue problem (Section A.10):

S_{B}^{x}ϕ_{i} = λ_{i}S_{T}^{x} ϕ_{i}, (i = 1, ⋯, n)  (8.33)

where λ_{i} = R(a) is an eigenvalue and ϕ_{i} the corresponding eigenvector of S_{B/T} = (S_{T}^{x})^{−1}S_{B}^{x}. Obviously, the transform vector a that maximizes R(a) is the eigenvector corresponding to the greatest eigenvalue λ_{1} = max{λ_{i} (i = 1, ⋯, n)}.

We next generalize the method above to the case of m > 1. The matrix form of the eigenequation above is:

S_{B}^{x}Φ = S_{T}^{x} ΦΛ  (8.34)

where Λ is the eigenvalue matrix of (S_{T}^{x})^{−1}S_{B}^{x}, and Φ = [ϕ_{1}, ⋯, ϕ_{n}] the eigenvector matrix (no longer orthogonal in general). This generalized eigenvalue problem can be solved by finding the matrix Φ that diagonalizes both S_{T}^{x} and S_{B}^{x} at the same time:

\{ \begin{matrix}S_{B}^{(y)} = Φ^{T} S_{B}^{x}Φ = Λ \\ S_{T}^{(y)} = Φ^{T} S_{T}^{x} Φ = I \end{matrix}  (8.35)

Left multiplying the inverse of the second equation to the first, we get

S_{B/T}^{(y)} = (S_{T}^{(y)})^{−1}S_{B}^{(y)} = Λ  (8.36)

The optimal transform matrix is composed of the eigenvectors A = [ϕ_{1}, ⋯, ϕ_{m}] corresponding to the m greatest of all n eigenvalues λ_{1} ≥ ⋯ ≥ λ_{m} ≥ ⋯ ≥ λ_{n}, so that – the signal components in y = [y_{1}, ⋯, y_{m}]^{T} are completely decorrelated,

each component y_{i} = ϕ_{i}^{T} x carries certain separability information s_{B}^{(y_{i})}/s_{T}^{(y_{i})} = λ_{i} (i = 1, ⋯, m), independent of others;

– the total separability contained in the m-D space, as the sum of the m greatest eigenvalues ∑_{i=1}^{m} λ_{i}, is maximized.

9 Principal Component Analysis

Principal Component Analysis

9.1 Covariance and Correlation

In principal component analysis (PCA), each pattern x is treated as a random vector of which each component x_{i} is a random variable with mean and variance

μ_{i} = E[x_{i}] = ∫ x_{i} p(x_{i}) dx_{i}
σ_{i}^{2} = E[(x_{i} − μ_{i})^{2}] = E[x_{i}^{2}] − μ_{i}^{2} = ∫ x_{i}^{2} p(x_{i}) dx_{i} − μ_{x_{i}}^{2}, (i = 1, ⋯, d)(9.1)

and the covariance of x_{i} and x_{j} is:

σ_{ij}^{2} = E[(x_{i} − μ_{i})(x_{j} − μ_{j})] = E[x_{i}x_{j}] − μ_{i}μ_{j}
= ∫ ∫ x_{i}x_{j} p(x_{i}, x_{j}) dx_{i} dx_{j} − μ_{i}μ_{j}  (i, j = 1, ⋯, d)  (9.2)

then the mean vector and covariance matrix of x are:

m_{x} = E[x] = [ μ_{1} ; ⋮ ; μ_{d} ]

Σ_{x} = E[(x − m_{x})(x − m_{x})^{T}] = E[xx^{T}] − m_{x}m_{x}^{T} = [ \begin{matrix}σ_{11}^{2} & ⋯ & σ_{1d}^{2} \\ ⋮ & ... & ⋮ \\ σ_{d1}^{2} & ⋯ & σ_{dd}^{2} \end{matrix} ](9.3)

Usually the joint probability density function p(x) = p(x_{1}, ⋯, x_{d}) of the random vector x is unknown. In this case, the mean vector m_{x} and covariance matrix Σ_{x} of x can be estimated by the method of maximum likelihood estimation (MLE) (Section 3.3.2) based on a set of observed data samples X = [x_{1}, ⋯, x_{N}]:

m̂_{x} = \frac{1}{N} ∑_{n=1}^{N} x_{n},  Σ̂ = \frac{1}{N} ∑_{n=1}^{N}(x_{n} − m̂)(x_{n} − m̂)^{T} = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T} − m̂m̂^{T}

(9.4)

Note that the rank of the d × d estimated covariance matrix Σ̂ is at most N − 1, due to the N samples in the dataset X, assumed to be are independent, and the additional constraint:

∑_{n=1}^{N}(x_{n} − m̂) = ∑_{n=1}^{N} x_{n} − Nm̂ = 0  (9.5)

The variance σ_{i}^{2} = E[(x_{i} − μ_{x_{i}})^{2}] can be treated as the dynamic energy contained in x_{i}, or the amount of information carried by x_{i}, while the trace trΣ_{x} = ∑_{i=1}^{d} σ_{i}^{2} can be considered as the total amount of dynamic energy contained in x. Also, the covariance σ_{ij}^{2} = E[(x_{i} − μ_{x_{i}})(x_{j} − μ_{x_{j}})] can be considered as the mutual energy, a measure of the correlation between x_{i} and x_{j}. By normalizing the covariance σ_{ij}^{2}, we get the correlation coefficient between x_{i} and x_{j}:

r_{ij} = \frac{σ_{ij}^{2}}{\sqrt{σ_{i}^{2} σ_{j}^{2}}} = \frac{σ_{ij}^{2}}{σ_{i} σ_{j}}  (9.6)

that measures how the two random variables x_{i} and x_{j} are correlated.

• r_{ij} = ±1 or |r_{ij}| = 1: x_{i} and x_{j} are maximally correlated. The information contained in the two variables is completely redendant, given the value of one of them, the value of the other is known.

• −1 < r_{ij} < 1 or |r_{ij}| < 1: x_{i} and x_{j} are correlated to different extents. The information they each carry has certain redendancy.

• r_{ij} = 0: x_{i} and x_{j} are uncorrelated. They each carry their own independent information with no redencdancy.

We see that the orrelation coefficient r_{ij} that measures how much two random variables are correlated also measures the redendancy of the information they carry. When there exists some data redendancy in the data, it is possible to carry out data compression by some method such as the principal component analysis based on the covariance Σ to reduce the data redendancy, so that the data size can be significantly reduced while the information (dynamic energy) contained in the data is still mostly preserved.

Examples

Six normally distributed 2-D datasets are generated with zero mean and the following covariance matrices:

Σ_{1} = [ \begin{matrix}1.0 & 0.95 \\ 0.95 & 1 \end{matrix} ] , Σ_{2} = [ \begin{matrix}1.0 & −0.95 \\ −0.95 & 1.0 \end{matrix} ] , Σ_{2} = [ \begin{matrix}1.0 & 0.7 \\ 0.7 & 1.0 \end{matrix} ]

Σ_{4} = [ \begin{matrix}1.0 & −0.7 \\ −0.7 & 1.0 \end{matrix} ] , Σ_{5} = [ \begin{matrix}3 & 0 \\ 0 & 1 \end{matrix} ] , Σ_{6} = [ \begin{matrix}1 & 0 \\ 0 & 3 \end{matrix} ]

These data points are plotted below, together with the correlation coefficient on top of each dataset.

9.2 Karhunen-Loeve Transformation

A data point in a d-dimensional space is represented by a vector x = [x_{1}, ⋯, x_{d}]^{T}, of which the components are the coordinates along the d standard orthonomal basis vectors {e_{1}, ⋯, e_{d}} that spann the space:

x = [ x_{1} ; x_{2} ; ⋮ ; x_{d} ] = x_{1} [ 1 0 ⋮ 0 ] + x_{2} [ 0 1 ⋮ 0 ] + ⋯ + x_{d} [ 0 ⋮ 0 1 ] = ∑_{i=1}^{d} x_{i}e_{i}

= [ e_{1} ⋯ e_{d} ] [ x_{1} ; ⋮ ; x_{d} ] = Ix  (9.7)

where I = [e_{1}, ⋯, e_{d}] is the identity matrix.

The space can also be spanned by any other orthonormal basis {a_{1}, ⋯, a_{d}} satisfying

a_{i}^{T} a_{j} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (9.8)

and the same vector x can also be represented as a weighted vector sum of these basis vectors:

x = ∑_{i=1}^{d} y_{i}a_{i} = [ a_{1} ⋯ a_{d} ] [ y_{1} ; ⋮ ; y_{d} ] = Ay  (9.9)

where A = [a_{1}, ⋯, a_{d}] composed of d orthonormal column vectors is an orthogonal matrix satisfying AA^{T} = A^{T} A = I, and y = [y_{1}, ⋯, y_{d}]^{T} is a vector containing all d coordinates in the directions of the basis vector {a_{i}, ⋯, a_{d}}, which can be found by premultiplying A^{T} = A^{−1} on both sides of the equation above Ay = x:

A^{T} Ay = y = [ y_{1} ; ⋮ ; y_{d} ] = A^{T} x = [  a_{1}^{T} ; ⋮ ; a_{d}^{T}  ] x = [ a_{1}^{T} x ; ⋮ ; a_{d}^{T} x ]  (9.10)

of which the ith coordinate y_{i} = a_{i}^{T} x is the projection of x onto the ith basis vector a_{i}.

The basis vectors in A = [a_{1}, ⋯, a_{d}] can be considered as a rotated version of the standard basis in I = [e_{1}, ⋯, e_{d}], and the norm or length of x before and and y after the transform remain the same:

||x||^{2} = x^{T} x = (∑_{i=1}^{d} y_{i}a_{i})^{T} (∑_{j=1}^{d} y_{j}a_{j}) = ∑_{i=1}^{d} ∑_{j=1}^{d} y_{i}y_{j} a_{i}^{T} a_{j} = ∑_{i=1}^{d} y_{i}^{2} = ||y||^{2}

(9.11)

This identity can be considered as the generalized version of Parseval’s identity in the Fourier transform, as a special form of unitary transform (complex version of orthogonal transform).

Summarizing the above, we can define an orthogonal transform based on any orthogonal matrix A:

\{ \begin{matrix}y = A^{T} x & forward transform \\ x = Ay & inverse transform \end{matrix}  (9.12)

by which any vector x under the implicit standard basis, the column vectors of I = [e_{1}, ⋯, e_{d}], is transformed into y = A^{T} x under the explicit basis, the column vectors of A = [a_{1}, ⋯, a_{d}].

Any orthogonal transform y = A^{T} x is actually a rotation of the standard basis {e_{1}, ⋯, e_{d}} into another orthonormal basis {a_{1}, ⋯, a_{d}} spanning the same space, while x and y are just the coordinates or coefficients of the same vector under these two different coordinate systems.

If x is treated as a random vector, then the linear transform y = A^{T} x, is also a random vector, and its mean vector and covariance can be found as:

m_{y} = E[y] = E[A^{T} x] = A^{T} E[x] = A^{T} m_{x}
Σ_{y} = E[yy^{T}] − m_{y}m_{y}^{T} = E[A^{T} xx^{T} A] − A^{T} m_{x}m_{x}^{T} A
= A^{T} [E[xx^{T}] − m_{x}m^{T}]A = A^{T} Σ_{x}A  (9.13)

In particular, the Karhunen-Loeve Transform (KLT) is just one of such orthogonal transforms in the form of y = V^{T} x, where the orthogonal transform matrix V = [v_{1}, ⋯, v_{d}] is the eigenvector matrix of the covariance matrix Σ_{x} of x, composed of the d normalized eigenvectors of Σ_{x}. As in general the covariance matrix Σ_{x} is symmetric and positive definite, its eigenvalues {λ_{1}, ⋯, λ_{d}} are real and positive, and its eigenvectors are orthogonal, i.e., its eigenvector matrix V is indeed an orthogonal matrix satisfying V^{T} = V^{−1} or V^{T} V = I.

The d eigenvalues {λ_{1}, ⋯, λ_{d}} and the corresponding eigenvectors {v_{1}, ⋯, v_{d}} can then be found by solving the eigenequations:

Σ_{x}v_{i} = λ_{i}v_{i} (i = 1, ⋯, d)  (9.14)

which can be combined to be expressed in matrix form:

Σ_{x}V = Σ_{x}[v_{1}, ⋯, v_{d}] = [v_{1}, ⋯, v_{d}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{d} \end{matrix} ] = VΛ  (9.15)

where Λ = diag(λ_{1}, ⋯, λ_{d}) is a diagonal matrix. Premultiplying V^{T} = V^{−1} on both sides, we get

V^{T} Σ_{x}V = V^{−1}VΛ = Λ  (9.16)

Taking inverse on both sides, we also get

(V^{T} Σ_{x}V)^{−1} = V^{−1}Σ_{x}^{−1}(V^{T})^{−1} = V^{T} Σ_{x}^{−1}V = Λ^{−1}  (9.17)

We see that both Σ_{x} and its inverse Σ_{x}^{−1} are diagonalized by the orthogonal eigenvector matrix V to become Λ and Λ^{−1} respectively.

Based on the orthogonal eigenvector matrix V, the KLT is defined as:

y = [ y_{1} ; ⋮ ; y_{d} ] = V^{T} x = [  v_{1}^{T} ; ⋮ ; v_{d}^{T}  ] x = [ v_{1}^{T} x ; ⋮ ; v_{d}^{T} x ]  (9.18)

of which the ith component y_{i} = v_{i}^{T} x is the projection of x onto the ith eigenvector v_{i}. Premultiplying V = (V^{T})^{−1} on both sides of the forward transform y = V^{T} x, we get the inverse KLT transform, by which vector x is represented as a linear combination of the eigenvectors {v_{1}, ⋯, v_{d}} as the basis vectors:

x = Vy = [ v_{1} ⋯ v_{d} ] [ y_{1} ; ⋮ ; y_{d} ] = ∑_{i=1}^{d} y_{i}v_{i}  (9.19)

Example:

Σ_{1} = [ \begin{matrix}2.964 & 0.544 \\ 0.544 & 4.687 \end{matrix} ] , Σ_{2} = [ \begin{matrix}5.036 & 3.083 \\ 3.083 & 2.830 \end{matrix} ] , Σ_{3} = [ \begin{matrix}3.043 & −3.076 \\ −3.076 & 3.930 \end{matrix} ]

(9.20)

r_{1} = 0.146,  r_{2} = 0.817,  r_{3} = −0.889  (9.21)

λ_{1,2} = (4.845, 2.807), λ_{1,2} = (7.207, 0.659), λ_{1,2} = (6.594, 0.379)

(9.22)

9.3 Optimality of KLT

The KLT is the optimal orthogonal transform among all possible orthogonal transforms y = A^{T} x based on any orthogonal matrix A satisfying A^{T} A = I, in the following sense:

• KLT completely decorrelates the signal.

• KLT maximally compacts the energy (information) contained in the signal.

KLT Completely Decorrelates the Signal

The mean vector and covariance matrix of the random vector y = V^{T} x after the KLT transform can found as

m_{y} = E[y] = E[V^{T} x] = V^{T} E[x] = V^{T} m_{x}  (9.23)
Σ_{y} = E[(y − m_{y})(y − m_{y})^{T}] = E[V^{T} (x − m_{x})][V^{T} (x − m_{x})]^{T}
= V^{T} E [(x − m_{x})(x − m_{x})^{T} ] V = V^{T} Σ_{x}V = Λ
= [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{d} \end{matrix} ] = [ \begin{matrix}σ_{y_{1}}^{2} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & σ_{y_{d}}^{2} \end{matrix} ]  (9.24)

Based on the fact that the covariance matrix Σ_{y} is a diagnal matrix, we have the following two observations:

• The zero off-diagonal component σ_{ij}^{2} = 0 (i ≠ j) for the covariance between any two components y_{i} and y_{j} indicates they are completely decorrelated after the KLT, i.e., each component carries its own independent information;

• The nonzero diagonal component for the variance λ_{i} = σ_{i}^{2} = E[(y_{i}−μ_{y_{i}})^{2}] (i = 1, ⋯, c) of the ith component y_{i} represents the dynamic energy or amount of information contained in y_{i}. As Σ_{x} and Σ_{y} = Λ share the same eigenvalues {λ_{1}, ⋯, λ_{d}}, and therefore have the same trace

E_{x} = trΣ_{x} = ∑_{i=1}^{d} λ_{i} = trΣ_{y} = E_{y}  (9.25)

We see that the total amount of energy or information contained in the signal x before the KLT and y after the KLT remains the same, i.e., the energy or information contained in the signal is conserved by the KLT. This is generally true for all orthogonal transforms y = A^{T} x (such as the Fourier transform), due to the fact that the eigenvalues remain the same under any orthogonal transform B = R^{T} AR, i.e., matrices A and B share the same eigenvalues, and thereby the same trace and determinant.

KLT Optimally Compacts Signal Energy While the total energy contained in the signal is conserved by any orthogonal

transform, the distribution of this energy among the d compoents before and after the transform may be very different. We can show that the KLT is optimal in the sense that it redistributes the total energy in such a way that it is maximally compacted into a subset of components of y = V^{T} x.

Consider a generic orthorgonal transform y = A^{T} x based on an arbitary orthogonal matrix A = [a_{1}, ⋯, a_{d}] satisfying A^{T} = A^{−1}:

y = [ y_{1} ; ⋮ ; y_{d} ] = A^{T} x = [ a_{1}^{T} ; ⋮ ; a_{d}^{T}  ] x  (9.26)

We also have

y_{i} = a_{i}^{T} x,  μ_{y_{i}} = E(y_{i}) = a_{i}^{T} E(x) = a_{i}^{T} m_{x},  (i = 1, ⋯, d) (9.27)

We then consider the energy contained in d′ < d out of the d components of y = A^{T} x expressed as a function of the transform matrix A = [a_{1}, ⋯, a_{d}]:

E′(A) = ∑_{i=1}^{d′} σ_{y_{i}}^{2} = ∑_{i=1}^{d′} E [(y_{i} − μ_{y_{i}})^{2}] = ∑_{i=1}^{d′} E [a_{i}^{T} (x − m_{x_{i}}) (x − m_{x_{i}})^{T} a_{i}]

= ∑_{i=1}^{d′} a_{i}^{T} E [(x − m_{x_{i}})(x − m_{x_{i}})^{T} ] a_{i}^{T} = ∑_{i=1}^{d′} a_{i}^{T} Σ_{x}a_{i}  (9.28)

We can find the optimal orthogonal transform matrix A that maximizes this partial energy E′ by solving the following optimization problem with the constraint that the column vectors of A are normalized with a_{j}^{T} a_{j} = 1:

\{ \begin{matrix}maximize: E′(A) \\ subject to: a_{j}^{T} a_{j} = 1 (j = 1, ⋯, d′) \end{matrix}  (9.29)

The Lagrange function of this constrained optimization problem is

L(A) = E′(A) − ∑_{j=1}^{d′} λ_{j}(a_{j}^{T} a_{j} − 1)  (9.30)

To find the optimal A, we set partial derivative with respect to each a_{i} (i = 1, ⋯, d′) to zero:

\frac{∂}{∂a_{i}} L(A) = \frac{∂}{∂a_{i}} [∑_{j=1}^{d′}(a_{j}^{T} Σ_{x}a_{j} − λ_{j}a_{j}^{T} a_{j} + λ_{j})]

= \frac{∂}{∂a_{i}} (a_{i}^{T} Σ_{x}a_{i} − λ_{i}a_{i}^{T} a_{i}) = 2Σ_{x}a_{i} − 2λ_{i}a_{i} = 0  (9.31)

The resulting equation Σ_{x}a_{i} = λ_{i}a_{i} happens to be the eigenequation of the covariance matrix Σ_{x}, i.e., the column vectors of the optimal transform matrix A must be the eigenvectors of Σ_{x} satisfying:

Σ_{x}a_{i} = λ_{i}a_{i}, i.e. a_{i}^{T} Σ_{x}a_{i} = λ_{i}, (i = 1, ⋯, d′)  (9.32)

We therefore see that the optimal transform is indeed the KLT transform

A = [a_{1}, ⋯, a_{d}] = V = [v_{1}, ⋯, v_{d}]  (9.33)

and

E′(V) = ∑_{i=1}^{d′} v_{i}^{T} Σ_{x}v_{i} = ∑_{i=1}^{d′} λ_{i} ≥ ∑_{i=1}^{d} λ_{i} = E_{y} = E_{x}  (9.34)

This partial energy E′(V) is maximized if we choose those eigenvectors v_{i} corresponding to the d′ greatest eigenvalues of Σ_{x}: λ_{1} ≥ λ_{2} ≥ ⋯ λ_{d′} ⋯ ≥ λ_{d}.

9.4 Geometric Interpretation of the KLT

The optimality of the KLT discussed above can be demonstrated geometrically, based on the assumption that the random vector x = [x_{1}, ⋯, x_{d}]^{T} has a normal probability density function:

p(x) = N (x, m_{x}, Σ_{x}) = \frac{1}{(2π)^{d/2} |Σ_{x}|^{1/2}} exp [−\frac{1}{2}(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x})]

(9.35)

with mean vector m_{x} and covariance matrix Σ_{x}. The shape of this normal distribution in the d-dimensional space can be represented by the iso-hypersurface in the space determined by the equation

N (x, m_{x}, Σ_{x}) = c_{0}  (9.36)

where c_{0} is some constant. This equation can be converted into an equivalent equation:

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = c_{1}  (9.37)

where c_{1} is another constant related to c_{0}. As Σ^{−1} is positive definite as well as Σ, this equation represents an hyper ellipsoid in the d-dimensional space. In particular, when d = 2, x = [x_{1}, x_{2}]^{T}, with positive definite Σ_{x}^{−1}:

Σ_{x}^{−1} = [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ] and |Σ_{x}^{−1}| = AC − B^{2}/4 > 0  (9.38)

then the quadratic equation above becomes

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = [x_{1} − μ_{x_{1}}, x_{2} − μ_{x_{2}}] [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ] [ \begin{matrix}x_{1} − μ_{x_{1}} \\ x_{2} − μ_{x_{2}} \end{matrix} ]

= A(x_{1} − μ_{x_{1}})^{2} + B(x_{1} − μ_{x_{1}})(x_{2} − μ_{x_{2}}) + C(x_{2} − μ_{x_{2}})^{2} = c_{1}

representing an ellipse (instead of other quadratic curves such as hyperbola and parabola) centered at m_{x} = [μ_{1}, μ_{2}]^{T}. When d = 3, the quadratic equation represents an ellipsoid. In general when d > 3, the equation N (x, m_{x}, Σ_{x}) = c_{0} represents a hyper-ellipsoid in the d-dimensional space.

Substituting x = Vy and m_{x} = Vm_{y} into the iso-surface equation Eq. (9.37), we get the equation for the hyper-ellipsoid after the KLT y = V^{T} x:

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = [V(y − m_{y})]^{T} Σ_{x}V(y − m_{y})
= (y − m_{y})^{T} V^{T} Σ_{x}^{−1}V(y − m_{y}) = (y − m_{y})^{T} Σ_{y}^{−1}(y − m_{y})
= (y − m_{y})^{T} Λ^{−1}(y − m_{y}) = ∑_{i=1}^{d} \frac{(y_{i} − μ_{y_{i}})^{2}}{λ_{i}} = ∑_{i=1}^{d} \frac{(y_{i} − μ_{y_{i}})^{2}}{σ_{y_{i}}^{2}} = c_{1}(9.39)

This a standardized hyper-ellipsoid. We therefore see that the KLT transform y = V^{T} x is actually a rotation of the coordinate system of the d-dimensional space, which is spanned by the standard basis {e_{1}, ⋯, e_{d}} before the KLT and the eigenvectors {v_{1}, ⋯, v_{d}} as the basis vectors after the KLT.

As the result, the principal semi-axes of the ellipsoid representing the Gaussian distribution of the dataset become in parallel with the axes of the new coordinate system, i.e., the ellipsoid becomes standardized. Moreover, the length of the ith principal semi-axis is proportional to the standard deviation σ_{y_{i}} = \sqrt{λ_{i}} of the ith variable y_{i}. This is the reason why KLT possesses the two desirable properties: (a) the decorrelation of the signal components, and (b) redistribution and compaction of the energy or information contained in the signal, as illustrated in the figure below.

Examples

9.5 Computation of the KLT

Due to KLT’s properties of signal decorrelation and energy compaction, it can be used for data compression by reducing the dimensionality of the dataset. Specifically, we carry out the following steps:

1. Find the mean vector m_{x} and the covariance matrix Σ_{x} of the random vectors x based on the given dataset X = [x_{1}, ⋯, x_{N}] containing N samples of x.

2. Find the eigenvalues λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{d} of Σ_{x}, sorted in descending order, and their corresponding eigenvectors {v_{1}, v_{2}, ⋯, v_{d}}.

3. Determine the reduced dimensionality d′ < d so that the percentage of energy contained after the transform is no less than a given threshold (e.g., 95%):

\frac{∑_{i=1}^{d′} λ_{i}}{∑_{i=1}^{d} λ_{i}} = \frac{∑_{i=1}^{d′} σ_{y_{i}}^{2}}{∑_{i=1}^{d} σ_{y_{i}}^{2}} ≥ 0.95  (9.40)

4. Construct an d × d′ transform matrix composed of the d eigenvectors corresponding to the d greatest eigenvalues of Σ_{x}:

V′ = [v_{1}, ⋯, v_{d′}]_{d×d′}  (9.41)

and carry out the KLT based on V′:

y = [ y_{1} ; ⋮ ; y_{d′} ]_{d′×1} = V^{′T} x = [  v_{1}^{T} ; ⋮ ; v_{d′}^{T}  ]_{d′×d} [ x_{1} ; ⋮ ; ⋮ ; x_{d} ]_{d×1}

(9.42)

This is a lossy compression by which the dimensionality is reduced from d to d′ < d, with the following error representing the percentage of information lost:

∑_{i=d′+1}^{d} λ_{i}/ ∑_{i=1}^{d} λ_{i}  (9.43)

But as λ_{d′+1}, ⋯, λ_{d} are the smallest d − d′ eigenvalues, the error is minimum (e.g., 5%).

5. Reconstruct x by inverse KLT:

x = [ x_{1} ; ⋮ ; ⋮ ; x_{d} ]_{d×1} = V′y = [ v_{1} ⋯ v_{d′} ]_{d×d′} [ y_{1} ; ⋮ ; y_{d′} ]_{d′×1} = ∑_{i=1}^{d′} y_{i}v_{i}

(9.44)

The rank of the d × d estimated covariance matrix Σ̂_{x} in Eq. (9.4) based on the dataset X = [x_{1}, ⋯, x_{N}] is no greater than N − 1. When N ≤ d, then the rank of Σ̂ is at most N − 1 < d, i.e., there are no more than N − 1 non-zero eigenvalues while all remaining d − (N − 1) = d − N + 1 eigenvalues are zero. Also, if the dimensionality d ≫ N is high, the computational complexity O(d^{3}) for solving this eigenvalue problem of Σ̂_{x} may be expensive. In this case, we can find the N − 1 non-zero eigenvalues and their corresponding eigenvectors of Σ̂_{x} by solving the eigenvalue problem of an N × N matrix with much lower complexity of O(N^{3}).

Specifically, for convenience and without loss of generality, we assume that the data have zero mean m_{x} = 0, so that the covariance matrix of the given dataset X can be estimate as

Σ̂_{x} = \frac{1}{N} ∑_{n=1}^{N}(x_{n}−m_{x})(x_{n}−m_{x})^{T} = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T} = \frac{1}{N} [x_{1}, ⋯, x_{N}] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = \frac{1}{N} XX^{T}

and the eigenequation of this d × d matrix can be written as

Σ̂_{x}v_{i} = \frac{1}{N} XX^{T} v_{i} = λ_{i}v_{i}  (9.45)

Pre-multiplying X^{T} on both sides, we get the eigenequation of an N × N matrix X^{T} X/N:

\frac{1}{N} X^{T} X (X^{T} v_{i}) = λ_{i} (X^{T} v_{i}) i.e. \frac{1}{N} X^{T} Xu_{i} = λ_{i}u_{i}  (9.46)

where u_{i} = X^{T} v_{i} is the eigenvector of X^{T} X/N. We see that XX^{T} /N and X^{T} X/N share the same non-zero eigenvalues. When N < d, we can solve the eigenvalue problem in Eq. (9.46) with complexity O(N^{3}), instead that of the original problem in Eq. (9.45) with O(d^{3}), to get the N − 1 non-zero eigenvalues λ_{i}, (i = 1, ⋯, N − 1), and the corresponding eigenvectors u_{i} = X^{T} v_{i}. To find the eigenvectors v_{i} of the original matrix Σ̂_{x} = XX^{T} /N, we pre-multiply X on both sides of the eigenequation above and get

\frac{1}{N} XX^{T} (Xu_{i}) = λ_{i} (Xu_{i})  (9.47)

Comparing this with Eq. (9.45), we see that Xu_{i} = XX^{T} v_{i} = Nλ_{i}v_{i} is proportional to the original eigenvector v_{i} corresponding to λ_{i}, and they become the same when normalized.

9.6 Comparison with Other Orthogonal Transforms

Here we compare the KLT, in terms of signal decorrelation and energy compaction, with other orthogonal transforms such as discrete cosine transform (DCT) and Fourier transform (DFT), as well as identity transform IT (no transform), based on two images of cloud and sand as shown below.

We treat each colume of the image as an observation (instantiation) of a 1-D random vector x, and apply an orthogonal transforms y = A^{T} x to x based on the covariance matrix Σ_{x}, and compare the corresponding covariance matrices Σ_{y} after the transform to see how well each transform decorrelates the signal and compacts its energy.

The figure below shows the original images (left) and the covariance matrices in image form (pixel values proportional to the values in the covariance matrix) after three orthogonal transforms of IT, DCT, and KLT.

• Sand image

In the second panel showing the covariance after IT, the off-diagonal pixels are dark, indicating the pixels in the original image are not hithly correlated at all. In the third panel, the covariance after DCT looks similar to that before the transform, indicating that the DCT has little effect in terms of signal decorrelation and energy compaction. Finally, in the rightmost panel showing the covariance after KLT, all off diagonal elements are zero, i.e., the signal components are completely decorrelated.

• Cloud image

In the second panel for the covariance after IT, there exist some bright areas off the diagonal, indicating that many signal components close to each other are highly correlated. In the third panel for the covariance after DCT, the values of the off diagonal elements are significantly reduced, indicating that the signal components are significantly decorrelated. Finally, in the right-most panel showing the covariance after KLT, all off diagonal elements are zero, i.e., the signal components are completely decorrelated.

The effect of energy redistribution and compaction of these different transforms are also shown in the figure below, where the variances of signal components are sorted and plotted.

We see that due to the physical nature of the clouds and sand grains, the textures in the corresponding images are very different. In the cloud image, the neighboring pixels are highly correlated, while in the sand image, they are not much correlated at all. Correspondingly, the DCT can effectively decorrelate the cloud image, but much less so in the sand image. But in either case, whether the pixels in the image are highly correlated or not, the KLT can always completely decorrelate the signal and optimally compact the energy.

The table below further demonstrate the effect of energy compaction quantitatively in terms of the number of components out of the total of 256 components needed after the transform in order to keep a certain percentage of the total dynamic energy (information). For example, if it is desired to keep 95% of the total energy contained in the original signal, 230 components are needed with no transform, 22 are needed after DCT, and only 13 are needed after KLT. Note that DCT’s performance is reasonably close to that of the optimal KLT in this case.

Percentage: 90% 95% 99% 100%
IT:  209 230 250 256
DCT:  10 22 97 256
KLT:  7 13 55 256

(9.48)

Although KLT is optimal in terms of signal decorrelation and energy compaction, it is not as convenient as other transforms for two reasons. First, the KLT is data-specific, i.e., the transform matrix is the eigenvector matrix of the covariance matrix of the dataset, which needs to be estimated based on sufficient amount of data samples, while all other orthogonal transforms are data indepenent. Seond, the computational cost of the KLT is significantly higher than that of other transforms, due to the O(d^{3}) complexity for solving the eigenvalue problem of the d × d matrix covairance matrix Σ_{x}, while for most other popular orthogonal transforms fast algorithm exist with O(d log_{2} d) complexity instead of O(d^{2}) required by the KLT.

9.7 Application to Image Data

The KLT can be applied to a set of N images for various purposes such as data compression and feature extraction. There are two alternative ways to carry out the KLT on the N images each containing K pixels, depending on how the random vector x is defined based on the image data which can be represented in the form of an N × K 2-D array.

• A K-dimensional vector x_{n} can be formed by all K pixels (by concatenating the rows or columns) of each of the N images. These N vectors each for one of the N images are represented by a K × N data array X = [x_{1}, ⋯, x_{N}], and treated as the samples of a K-dimensional random vector x, its covariance matrix can be estimated (assuming with zero mean) as:

Σ = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T} = \frac{1}{N} [x_{1}, ⋯, x_{N}] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = \frac{1}{N} (XX^{T} )_{K×K} (9.49)

• An N-dimensional vector can be formed by the pixels at the same position (in the ith row and jth column of the image array) from all N images. There are K such vectors each for one of the K pixels in the images. These vectors are the rows of the X defined above, or the columns of X^{T}. The covariance matrix of these N dimensional vectors can be estimated as:

Σ′ = \frac{1}{K} (X^{T} X)_{N×N} .  (9.50)

As shown previously, these two different covariance matrices share the same eigenvalues. The eigenequations for X^{T} X and X^{T} X (with the constant coefficients 1/K and 1/N neglected) are:

X^{T} Xv = λv,  XX^{T} u = μu.  (9.51)

Pre-multiplying X^{T} on both sides of the second equation we get

X^{T} X[X^{T} u] = μ[X^{T} u].  (9.52)

which is actually the first eigenequation with the same eigenvalue μ = λ and eigenvector v = X^{T} u when normalized. The two covariance matrices Σ and Σ′ have the same rank R = min(N, K) (if X is not degenerate) and therefore the same number of non-zero eigenvalues. Consequently, the KLT can be carried out based on either matrix with the same effects in terms of the signal decorrelation and energy compaction. As the number of pixels in the image is typically much greater than the number of images, K > N, we will take the second approach above to treat the pixels in the same position in all N images as a sample of an N-dimensional random signal vector and carry out the KLT based on the N ×N covariance matrix Σ̂.

We can now carry out the KLT to each of the K N-dimensianl vectors x corresponding to each pixel of the N images to obtain another N-dimensional vector y = v^{∗}x for the same pixel of a set of N eigen-images, as shown below. After the KLT, most of the energy/information contained in the N images, representing the variations among all N images, is concentrated in the first few eigen-images corresponding to the greatest eigenvalues, while the remaining eigen-images can be omitted without losing much energy/information. This is the foundation for various KLT-based image compression and feature extraction algorithms. The subsequent operations such as image recognition and classification can all be carried out in a much lower dimensional space.

We now consider some of such applications.

• Remote sensing data compression

In remote sensing, images of the surface of either the Earth or other planets such as Mars are taken by a multispectral camera system on board satellite, for various studies (e.g., geology, geography, etc.). The camera system has an array of N sensors, typically a few tens or even over a hundred, each sensitive to a different wavelength band in the visible and infrared range of the electromagnetic spectrum. Depending on the number of sensors, the data are referred to as either multi or hyper-spectral images.

These sensors will produce a set of N images covering the same surface area on the ground. For the same position in these images, there are N pixel values each from one wavelength band representing the spectral profile that characterizes the material on the surface area corresponding to the pixel. A typical application of the multi or hyper-spectral image data is to classify the pixels into different types of materials (different types of rocks, vegetations, polutions, etc.). When N is large, KLT can be used to reduce the dimensionality without loss of much information. Specifically, we consider the N values associated with each pixel form an N-dimensional random vector, and then carry out KLT to reduce its dimensionality. All classification can be carried out in this low dimensional space, thereby significantly reducing the computational complexity.

• Video image compression

A sequence of N = 8 frames of a video of a moving escalator and their eigen-images are shown respectively in the upper and lower parts of the figure below.

It is interesting to observe that the first eigen-image corresponding to the greatest eigenvalue (left panel of the third row of the figure) represents mostly the static scene of the image frames representing the main variations in the image (carrying most of the energy), while the subsequent eigenimages represent mostly the motion in the video, the variation between the frames. For example, the motion of the people riding on the escalator is mostly reflected by the first few eigen-images following the first one, while the motion of the escalator stairs is mostly reflected in the subsequent eigen-images.

The 8×8 covariance matrix and the energy distribution among the eight components plot before and after the KLT are shown below.

We see that due to the spatial correlation between nearby pixels, the covariance matrix before the KLT (left) can be modeled by the squared exponental function, while the covariance matrix after the KLT (middle) is completely decorrelated and the energy is highly compacted into a small number of principal components (here the first component), as also clearly shown in the comparison of the energy distribution before and after the KLT (right).

• Eigenface and face recognition

Twenty images of faces (N = 20):

The eigen-images after KLT:

The table below shows the percentage of energy contained in each of the components:

components  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19  20
percentage energy  48.5 11.6 6.1 4.6 3.8 3.7 2.6 2.5 1.9 1.9 1.8 1.6 1.5 1.4 1.3 1.2 1.1 1.1 0.9  0.8

accumulative energy 48.5 60.1 66.2 70.8 74.6 78.3 81.0 83.5 85.4 87.3 89. 90.7 92.2 93.6 94.9 96.1 97.2 98.2 99.2 100.0

(9.53)

Reconstructed faces using 95% of the total information (15 out of 20 components):

• Hand-written number recognition

The goal here is to recognize hand-written number from 0 to 9 in an image, as those in the figure below, containing N = 2240 samples for the K = 10 numbers. Each sample in the 16 × 16 image can be represented by a d = 16 × 16 = 256 dimensional vector by concatenating all columns (or rows) of the image. The KLT can be carried out to significantlly reduce the dimensionality of the vectors from d = 256 to some d′ ≪ d, based on either the covariance matrix Σ_{x} = S_{T} of all N = 2240 sample vectors representing the over all distribution of these data points, or the betweenclass scatter matrix S_{B} previously considered representing the separability of the ten classes.

Specifically, we use the eigenvectors corresponding to the d′ greatest eigenvalues of the covariance matrix or the between-class scatter matrix to form a d′ by d transform matrix. After the KLT transform by the data, certain classification algorithm can be carried out in the much reduced d’ dimensional space.

The energy distribution over all d = 256 signal components is plotted below for the original signal (top), after the KLT based on S_{B} (middle), and after the KLT based on S_{T}.

For the KLT based on S_{T} with rank N − 1, d′ = 79 components are needed to keep 95.1% of the total energy, in comparison to the KTL based on S_{B} with rank rank(S_{B}) = K −1 = 9, requiring are only d′ = 9 principal components corresponding to the same number of non-zero eigenvalues of S_{B} to keep 100% of the total energy representing the separability information. The percentage energy conteined in these non-zero eigenvalues are: 28.6%, 22.7%, 14.4%, 10.9%, 7.1%, 7.0%, 3.8%, 3.2%, 2.4%.

The corresponding d = 256 dimensional eigenvectors can be visualized when converted to 16×16 eigenimages, representing the basis by which any original images can be represented as a linear combination of such eigenimages, as shown in the figure below. The 10th eigenimage corresponding to a zero eigenvalue contains only some random noise.

If we only keep the first two or three principal components (corresponding to the greatest eigenvalues) after the KLT, the dataset can be visualized as shown in the figure below. The sample points in each of the ten different classes are color-coded. It can be seen that even when the dimensions are much reduced from N = 256 to d = 3 or even d = 2, it is still possible to separate the ten classes reasonably well.

10 Variations of PCA

Variations of PCA

10.1 Kernel Methods

In the kernel method, all data points as a vector x = [x_{1}, ⋯, x_{d}]^{T} in the original d-dimensional feature space are mapped by a kernel function z = ϕ(x) into a higher, possibly infinite, dimensional space

x ⟹ z = ϕ(x)  (10.1)

The presumption of the method is that the data points only appear in the form of inner product x_{m}^{T}x_{n} in the algorithm, then based on the kernel trick, the kernel function z = ϕ(x) never needs to be actually carried out. In fact, the form of the kernel function ϕ(x) and the dimensionality of the higher dimensional space do not need to be explicitly specified or known.

The motivation for such a kernel mapping is that the relevant operations such as classification and clustering may be carried out much more effectively once the dataset is mapped to the higher dimensional space. For example, classes not linearly separable in the original d-dimensional feature space may be trivially separable in a higher dimensional space, as illustrated by the following examples.

Example 1

In 1-D space, classes C_{−} = {x|(a ≤ x ≤ b)} and C_{+} = {x|(x ≤ a) or (x ≥ b)} are not linearly separable. By the following mappping from 1-D space to 2-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \end{matrix} ] = [ \begin{matrix}x \\ (x − (a + b)/2)^{2} \end{matrix} ]  (10.2)

the two classes can be separated by a threshold in the second dimension of the 2-D space.

Example 2:

The method above can be generalized to higher dimensional spaces such as mapping from 2-D to 3-D space. Consider two classes in a 2-D space that are not linearly separable: C_{−} = {x, ||x|| < D} and C_{+} = {x, ||x|| > D}. However, by the following mappping from 2-D space to 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}^{2} + x_{2}^{2} \end{matrix} ]  (10.3)

the two classes can be trivially separated linearly by thresholding in the third dimension of the 3-D space.

Example 3:

In 2-D space, in the exclusive OR dataset, the two classes of C_{−} containing points in quadrants I and III, and C_{+} containing points in quadrants II and IV are not linearly separable. However, by mapping the data points to a 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}x_{2} \end{matrix} ]  (10.4)

the two classes can be separated by simply thresholding in the third dimension of the 3-D space.

Definition: A kernel is a function that takes two vectors x_{m} and x_{n} as arguments and returns the inner product of their images z_{m} = ϕ(x_{m}) and z_{n} = ϕ(x_{n}):

K(x_{m}, x_{n}) = ϕ(x_{m})^{T} ϕ(x_{n}) = z_{m}^{T}z_{n}  (10.5)

The kernel function takes as input some two vectors x_{m} and x_{n} in the original feature space, and returns a scalor value as the inner product z_{m} and z_{n} in some higher dimensional space. If the data points in the original space only appear in the form of inner product, then the kernel function z = ϕ(x), called the kernel-induced implicit mapping, never needs to be explicitly specified, and the dimension of the new does not even need to be known.

The following is a set of commonly used kernel functions K(x, x′), which can be represented as an inner product of two vectors z = ϕ(x) and z′ = ϕ(x′) in a higher dimensional space.

• linear kernel (no kernel mapping)

Assume x = [x_{1}, ⋯, x_{d}]^{T}, x′ = [x′_{1}, ⋯, x′_{d}]^{T},

K(x, x′) = x^{T} x′ = ∑_{i=1}^{d} x_{i}x′_{i}  (10.6)

• polynomial kernels The binomial theorem states:

(x + y)^{n} = ∑_{k=0}^{n} ( \begin{matrix}n \\ k \end{matrix} ) x^{n−k}y^{k} = ∑_{k=0}^{n} \frac{n!}{k!(n − k)!} x^{n−k}y^{k}  (10.7)

where the binomial coefficient

( \begin{matrix}n \\ k \end{matrix} ) = \frac{n!}{k!(n − k)!}  (10.8)

is the number of ways to distribute n items into two bins (k in one and n−k in the other). This result can be generalized to the multinormial case:

(x_{1}+⋯+x_{d})^{n} = ∑ ; ∑_{i=1}^{d} k_{i}=n ( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}} = ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!}x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}

(10.9)

where the multinomial coefficient

( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) = \frac{n!}{k_{1}! ⋯ k_{d}!}  (10.10)

is the number of ways to distribute n balls into d bins with k_{i} balls in the ith bin, and the summation is over all possible ways to get d non-negative integers k_{1}, ⋯, k_{d} that add up to n.

Now consider the homogeneous polynomial kernel for d-dimensional vectors x = [x_{1}, ⋯, x_{d}]^{T} defined as

K(x, x′) = (x^{T} x′)^{n} = (x_{1}x′_{1} + ⋯ + x_{d}x′_{d})^{n}
= ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}}) = ϕ(x)^{T} ϕ(x′) = z^{T} z′ (10.11)

where

z = ϕ(x) = [\sqrt{\frac{n!}{k_{1}! ⋯ k_{d}!}} (x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}) , (k_{i} ≥ 0, ∑_{i=1}^{d} k_{i} = n)]^{T} (10.12)

In particular, when d = 2 and n = 2, the polynomial kernel defined over 2-D vectors x = [x_{1}, x_{2}]^{T} is:

K(x, x′) = (x^{T} x′)^{2} = (x_{1}x′_{1}+x_{2}x′_{2})^{2} = (x_{1}x′_{1})^{2}+2x_{1}x′_{1}x_{2}x′_{2}+(x_{2}x′_{2})^{2} = ϕ(x)^{T} ϕ(x′) = z^{T} z′

(10.13)

where z = ϕ(x) = [x_{1}^{2}, \sqrt{2}x_{1}x_{2}, x_{2}^{2}] is a mapping from x in 2-D space to z in 3-D space.

A non-homogeneous polynomial kernel is defined as

K(x, x′) = (1 + x^{T} x′)^{n}  (10.14)

• The radial basis function (RBF) kernel The RBF kernel is defined as

K(x, x′) = e^{−||x−x′||^{2}/2σ^{2}} = e^{−γ||x−x′||^{2}}  (10.15)

where γ = 1/2σ^{2} is a parameter that can be adjusted to fit each specific dataset. This kernel can be wriiten as the inner product of two infinite dimensional vectors (for simplicity, we assume σ = 1):

K(x, x′) = e^{−||x−x′||^{2}/2} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} e^{x^{T} x′} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} \frac{(x^{T} x′)^{n}}{n!}

= e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} [\frac{1}{n!} ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}})]

= ∑_{n=0}^{∞} ∑ ; ∑_{i=1}^{d} k_{i}=n (e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}) (e^{−||x′||^{2}/2} \frac{x_{1}^{′k_{1}} ⋯ x_{d}^{′k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}})
= ϕ(x)^{T} ϕ(x′) = z^{T} z′  (10.16)

where

z = ϕ(x) = [e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}, (n = 0, ⋯, ∞, ∑_{k=1}^{n} k_{i} = n)]^{T}

(10.17)

is a vector in an infinite dimensional space. In particular, when d = 1 we have

K(x, x′) = e^{−(x−x′)^{2}/2} = e^{−x^{2}/2} e^{−x^{′2}/2} e^{xx′} = e^{−x^{2}/2} e^{−x^{′2}/2} ∑_{n=0}^{∞} \frac{(xx′)^{n}}{n!}

= ∑_{n=0}^{∞}(e^{−x^{2}/2} x^{n}/\sqrt{n!}) (e^{−x^{′2}/2} x^{′n}/\sqrt{n!})  (10.18)

where z = ϕ(x) = [e^{−x^{2}/2} x^{n}/\sqrt{n!}, (n = 0, ⋯, ∞)]^{T} is a kernel function that maps a 1-D space into an infinite dimensional space.

10.2 Kernel PCA

In kernel PCA, the PCA method, which maps the dataset to a low dimensional space, and the kernel method, which maps the datad set to a high dimensional space, are combined to take advantage of both the high dimensional space (e.g., better separability) and the low dimensional space (e.g., computational efficiency).

Specifically, we assume the given dataset X = [x_{1}, ⋯, x_{N}] in the original ddimensional feature space is mapped into Z = [z_{1}, ⋯, z_{N}] in a high dimensional space. Correspoindingly, the inner product x_{m}^{T}x_{n} in the original space is mapped into an inner product in the high dimensional space:

z_{m}^{T}z_{n} = K(x_{m}, x_{n}) = k_{mn} (m, n = 1, ⋯, N)  (10.19)

Given N data points in the dataset X, we can further define a kernel matrix:

K = Z^{T} Z = [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] [z_{1}, ⋯, z_{N}] = [ \begin{matrix}z_{1}^{T} z_{1} & ⋯ & z_{1}^{T} z_{N} \\ ⋮ & ... & ⋮ \\ z_{N}^{T}z_{1} & ⋯ & z_{N}^{T}z_{N} \end{matrix} ] = [ \begin{matrix}k_{11} & ⋯ & k_{1N} \\ ⋮ & ... & ⋮ \\ k_{N1} & ⋯ & k_{NN} \end{matrix} ]

(10.20)

To carry out KLT in the high dimensional space, we need to estimate the covariance matrix of z = ϕ(x):

Σ_{z} = \frac{1}{N} ∑_{n=1}^{N}(z_{n} − m_{z})(z_{n} − m_{z})^{T} = \frac{1}{N} ∑_{n=1}^{N} z_{n}z_{n}^{T} − m_{z}m_{z}^{T}  (10.21)

We first assume the samples in Z = [z_{1}, ⋯, z_{N}] have a zero mean m_{z} = 0, which will be justified later, and get (same as Eq. (9.45)):

Σ_{z} = \frac{1}{N} ∑_{n=1}^{N}(z_{n}−m_{z})(z_{n}−m_{z})^{T} = \frac{1}{N} ∑_{n=1}^{N} z_{n}z_{n}^{T} = \frac{1}{N} [z_{1}, ⋯, z_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] = \frac{1}{N} ZZ^{T}

(10.22)

and its eigenequations:

Σ_{z}v_{n} = (\frac{1}{N} ZZ^{T} ) v_{n} = λ_{n}v_{n} (n = 1, 2, ⋯)  (10.23)

We then premultiply Z^{T} on both sides of the eigenequation above to get (same as Eq. (9.46)):

\frac{1}{N} Z^{T} Z (Z^{T} v_{n}) = \frac{1}{N} Ku_{n} = λ_{n} (Z^{T} v_{n}) = λ_{n}u_{n}  (10.24)

where u_{n} = Z^{T} v_{n}. Solving this eigenequation of the N × N matrix Z^{T} Z/N = K/N, we get its eigenvectors u_{n} = Z^{T} v_{n} (n = 1, ⋯, N), corresponding to the eigenvalues {λ_{1}, ⋯, λ_{N}}, which are the same as the non-zero eigenvalues of Σ_{z} = ZZ^{T} /N. The eigenvectors v_{n} of Σ_{z} in Eq. (10.23) can now be written as:

v_{n} = \frac{1}{λ_{n}N} Z (Z^{T} v_{n}) = \frac{1}{λ_{n}N} Zu_{n}  (10.25)

which can be further normalized by imposing the condition ||v_{n}||^{2} = 1:

||v_{n}||^{2} = v_{n}^{T} v_{n} = \frac{1}{(λ_{n}N)^{2}} (Zu_{n})^{T} (Zu_{n}) = \frac{1}{(λ_{n}N)^{2}} u_{n}^{T} Z^{T} Zu_{n}
= \frac{1}{(λ_{n}N)^{2}} u_{n}^{T} Ku_{n} = \frac{1}{λ_{n}N} u_{n}^{T} u_{n} = \frac{1}{λ_{n}N} ||u_{n}||^{2} = 1  (10.26)

We see that if the eigenvectors u_{n} of K are rescaled to satisfy ||u_{n}||^{2} = λ_{n}N, then the eigenvectors v_{n} of Σ_{z} are normalized to satisfy ||v_{n}||^{2} = 1. Now we can carry out PCA in the high dimensional space by

y = [v_{1}, ⋯, v_{N}]^{T} z = V^{T} z  (10.27)

of which each component y_{n} = v^{T} z is the projection of z = ϕ(x) onto one of the eigenvectors v_{n} of Σ_{z}:

y_{n} = z^{T} v_{n} = z^{T} (\frac{1}{λ_{n}N} Zu_{n}) = \frac{1}{λ_{n}N} z^{T} Zu_{n} = \frac{1}{λ_{n}N} z^{T} [z_{1}, ⋯, z_{N}]u_{n}
= \frac{1}{λ_{n}N} [z^{T} z_{1}, ⋯, z^{T} z_{N}] u_{n} = \frac{1}{λ_{n}N} [k_{1}, ⋯, k_{N}] u_{n} = \frac{1}{λ_{n}N} k^{T} u_{n},  (n = 1, ⋯, N) (10.28)

where we have defined k_{n} = z^{T} z_{n} and k = [k_{1}, ⋯, k_{N}]^{T}. Although neither z nor v_{n} = Zu_{n}/λ_{n}N is available, their inner product y_{n} = z^{T} v_{n} can still be obtained based on the kernel function k_{n} = z^{T} z_{n}. We see that all data points in the high dimensional space appear only in the form of an inner product in the equation above, as well as in the eigenequation Eq. (10.24), the kernel mapping z = ϕ(x) never needs to be explicitly carried out.

The discussion above is based on the assumption that the data in the high dimensional space have zero mean m_{z} = 0. This cannot be achived by subtracting the mean from the data to get z̃ = z − m_{z}, as neither z nor m_{z} is available without carrying out the kernel mapping z = ϕ(x). However, we can find the inner product of two zero-mean data points in the high dimensional space as

k̃_{mn} = z̃_{m}^{T}z̃_{n} = (z_{m} − m_{z})^{T} (z_{n} − m_{z}) = (z_{m} − \frac{1}{N} ∑_{k=1}^{N} z_{k})^{T} (z_{n} − \frac{1}{N} ∑_{l=1}^{N} z_{l})

= z_{m}^{T}z_{n} − z_{m}^{T} (\frac{1}{N} ∑_{l=1}^{N} z_{l}) − (\frac{1}{N} ∑_{k=1}^{N} z_{k}^{T} ) z_{n} + \frac{1}{N^{2}} ∑_{k=1}^{N} ∑_{l=1}^{N} z_{k}^{T} z_{l}

= k_{mn} − \frac{1}{N} ∑_{l=1}^{N} k_{ml} − \frac{1}{N} ∑_{k=1}^{N} k_{kn} + \frac{1}{N^{2}} ∑_{k=1}^{K} ∑_{l=1}^{N} k_{kl},  (m, n = 1, ⋯, N) (10.29)

The kernel matrix composed of all N × N such inner products can be written as:

K̃ = K − 1_{N}K − K1_{N} + 1_{N}K1_{N}  (10.30)

where

1_{N} = \frac{1}{N} [ \begin{matrix}1 & ⋯ & 1 \\ ⋮ & ... & ⋮ \\ 1 & ⋯ & 1 \end{matrix} ]_{N×N}  (10.31)

If we replace the kernel matrix K for z with non-zero mean in the algorithm considered above by K̃ for z̃ with zero mean, then the assumption that z has zero mean m_{z} becomes valid. In summary, here are the steps of the kernel PCA algorithm:

• Given the dataset X = {x_{1}, ⋯, x_{N}}, obtain K based on kernel mapping k_{mn} = k(x_{m}, x_{n}) for all m, n = 1, ⋯, N, and then find K̃, as in Eq. (10.30); • Solve eigenvalue problem of K̃ to get the eigenvalue λ_{n} and the corresponding

eigenvector u_{n} for all i = 1, ⋯, N;

• Project each data point z = ϕ(x) (implicit) onto the nth eigenvector v_{n} to get the principal component: y_{n} = z^{T} v_{n} = k^{T} u_{n}/λ_{n}N for each n = 1, ⋯, N, as in Eq. (10.28).

• Carry out whatever operation (e.g., feature selection, classification) in the subspace spanned by the first M < N principal components {y_{1}, ⋯, y_{M}} in the high dimensional feature space, containing most of the information.

Note that in kernel PCA methods, the complexity for solving the eigenvalue problem of the N × N kernel matrix K is O(N^{3}), instead of O(d^{3}) for the linear PCA based on the d × d covariance matrix Σ_{x}.

Shown below is a Matlab function for the kernel PCA algorithm which takes the dataset X = [x_{1}, ⋯, x_{N}] as input and generates the eigenvalues in d and the dataset in the transform space Y. The function Kernel(X,X) calculates the kernel matrix K.

function [Y d]=KPCA(X)
    [D N]=size(X);
    K=Kernel(X,X);                      % kernel matrix of data X
    one=ones(N)/N;
    Kt=K-one*K-K*one+one*K*one;         % Kernel matrix of data with zero mean
    [U D]=eig(Kt);                      % solve eigenvalue problem for K
    [d idx]=sort(diag(D),'descend');    % sort eigenvalues in descending order
    U=U(:,idx);
    D=D(:,idx);
    for n=1:N
        U(:,n)=U(:,n)/d(n)/N;
    end
    Y=(K*U)';
end

Examples:

Visualization of a 2-D dataset of four classes, linear plot on the left and kernel PCA (based on RBF kernel) on the right:

Visualization of a 256-D dataset of 10 classes of hand-written digits:

Visualization of the 4-D iris dataset in 3-D space spanned by the first 3 principal components of the linear and kernel PCA transforms:

10.3 Factor Analysis and Expectation Maximization

The method of factor analysis (FA) models a set of d observed manifest variables in x = [x_{1}, ⋯, x_{d}]^{T} as a linear combination of a set of d′ < d unobserved hiden latent variables or common factors in z = [z_{1}, ⋯, z_{d′}]^{T}, to explain and reveal the variability and dependency among the d observed variables, typically correlated, in terms of the latent variables, assumed to be independent and therefore uncorrelated. The method of FA is therefore considered as a method for dimensionality reduction.

Specifically, we assume each of the observed variables in x is a linear combination of the d′ factors in z

x_{i} = ∑_{j=1}^{d′} w_{ij}z_{j} + e_{i} = [w_{i1}, ⋯, w_{id′}] [ z_{1} ; ⋮ ; z_{d′} ] + e_{i},  (i = 1, ⋯, d) (10.32)

or in matrix form

x = [ x_{1} ; ⋮ ; ⋮ ; x_{d} ] = [ \begin{matrix}w_{11} & ⋯ & w_{1d′} \\ ⋮ & ... & ⋮ \\ ⋮ & ... & ⋮ \\ w_{d1} & ⋯ & w_{dd′} \end{matrix} ] [ z_{1} ; ⋮ ; z_{d′} ] + [ e_{1} ; ⋮ ; ⋮ ; e_{d} ] = Wz + e (10.33)

where W is a d × d′ factor loading matrix, and e_{i} is the noise associated with x_{i}. Also, for simplicity and without loss of generality, we assume the dataset has a zero mean. If W were available, the d′ factors in z can be found by solving this over-determined linear equation system of d equations but d′ < d unknowns by the least-squares method (with minimum squared error ||e||^{2}):

ẑ = W^{−}x = (W^{T} W)^{−1}W^{T} x  (10.34)

wheer W^{−} = (W^{T} W)^{−1}W^{T} is the left pseudo-inverse of W.

However, as W is unavailable, it needs to be estimated together with z at the same time, based on the given dataset X = [x_{1}, ⋯, x_{N}], typically N ≫ d. This can be done by the general method of expectation-maximization (EM), an iterative algorithm that maximizes the expectation of the likelihood (ML) or maximum a posteriori (MAP) estimate of some model parameters, widely used in machine learning.

Specifically, we treat both z and e as random vectors, and make the following assumptions:

• The latent variables in z are of zero mean, independent of each other, and with unity vairance:

m_{z} = E[z] = 0,  Σ_{z} = Cov[z] = E[zz^{T}] = I  (10.35)

and they are normally distributed:

p(z) = N (0, I)  (10.36)

• The noise components in e have zero mean, and they are independent of each other with a diagonal covariance matrix:

m_{e} = E[e] = 0, Σ_{e} = Cov[e] = E[ee^{T}] = Ψ = [ \begin{matrix}ψ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & ψ_{d} \end{matrix} ]

(10.37)

and they are also normally distributed:

p(e) = N (0, Ψ)  (10.38)

• The latent variables and the noise are independent of each other:

Σ_{ze} = Cov[z, e] = E[(z − m_{z})(e − m_{e}^{T})] = E[ze^{T}] = 0  (10.39)

The two matrices W and Ψ defined above as the parameters of the FA model are denoted by θ = {W, Ψ}.

Based on the assumptions above, we desire to find the conditional pdf p(z|x) of the latent variable z given the observed variable x. To do so, we first find the pdf of x = Wz + e, which, as a linear combination of the two normally distributed random vectors z and e, is also normally distributied with p(x) = N (m_{x}, Σ_{x}), where

m_{x} = E[x] = E[Wz + e] = WE[z) + E(e] = 0  (10.40)
Σ_{x} = Cov[x] = E[(x − m_{x})(x − m_{x}^{T})] = E[xx^{T}] = E [(Wz + e)(Wz + e)^{T} ]
= WE[zz^{T}]W^{T} − E[ez^{T}]W^{T} − WE[ze^{T}] + E[ee^{T}]
= WW^{T} + Ψ  (10.41)

As both m_{x} and Σ_{x} are based on the model parameter θ = {W, Ψ}, the pdf of x is conditional on θ:

p(x|θ) = N (m_{x}, Σ_{x}) = N (0, WW^{T} + Ψ)  (10.42)

The joint distribution p(z, x) is also Gaussian with a zero mean

m = E ([ \begin{matrix}z \\ x \end{matrix} ]) = [ \begin{matrix}m_{z} \\ m_{x} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ]  (10.43)

and a covariance Σ composed of four submatrices:

Σ = [ \begin{matrix}Σ_{zz} & Σ_{zx} \\ Σ_{xz} & Σ_{xx} \end{matrix} ]  (10.44)

where

Σ_{zz} = I
Σ_{xx} = WW^{T} + Ψ
Σ_{zx} = Σ_{zx}^{T} = E[(z − m_{z})(x − m_{x})^{T}] = E[zx^{T}] = E[z(Wz + e)^{T}]
= E[zz^{T}]W^{T} + E[ze^{T}] = IW^{T} + 0 = W^{T}  (10.45)

The normal distribution p(z, x|θ) can now be expressed as:

p(x, z|θ) = p ([ \begin{matrix}z \\ x \end{matrix} ]) = N (m, Σ) = N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{zz} & Σ_{zx} \\ Σ_{xz} & Σ_{xx} \end{matrix} ])
= N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}I & W^{T} \\ W & WW^{T} + Ψ \end{matrix} ])  (10.46)

Based on this joint pdf of both x and z, we can further find the desired conditional pdfs of both p(x|z) and p(z|x) (Section 3.1.3):

• p(x|z, θ) = N (m_{x|z}, Σ_{x|z}), with

\{ \begin{matrix}m_{x|z} & = & m_{x} + Σ_{xz}Σ_{zz}^{−1}(z − m_{z}) = Wz \\ Σ_{x|z} & = & Σ_{xx} − Σ_{xz}Σ_{zz}^{−1}Σ_{zx} = WW^{T} + Ψ − WW^{T} = Ψ \end{matrix} (10.47)

• p(z|x, θ) = N (m_{z|x}, Σ_{z|x}), with

\{ \begin{matrix}m_{z|x} & = & m_{z} + Σ_{zx}Σ_{xx}^{−1}(x − m_{x}) = W^{T} (WW^{T} + Ψ)^{−1}x = Bx \\ Σ_{z|x} & = & Σ_{zz} − Σ_{zx}Σ_{xx}^{−1}Σ_{xz} = I − W^{T} (WW^{T} + Ψ)^{−1}W = I − BW \end{matrix}

(10.48)

where we have defined

B = W^{T} (WW^{T} + Ψ)^{−1}  (10.49)

Note that while p(z) = N (0, I) has zero mean and diagonal covariance, the conditional distribution p(z|x, θ) = N (m_{z|x}, Σ_{z|x}) has non-zero mean m_{z|x} and non-diagonal covariance Σ_{z|x}.

The computational complexity for the inversion of the N ×N matrix WW^{T} + Ψ is O(N^{3}). However, by applying the Woodbury matrix identity (Section A.6):

(Ψ + WW^{T})^{−1} = Ψ^{−1} − Ψ^{−1}W(I + W^{T} Ψ^{−1}W)^{−1}W^{T} Ψ^{−1}  (10.50)

where Ψ as a diagonal matrix can be easily inverted, and I + W^{T} Ψ^{−1}W is an d′ × d′ matrix that can be inverted with complexity O(d^{′3}) ≪ O(N^{3}).

The model parameter θ = {W, Ψ} can now be estimated based on the given dataset X by the EM algorithm, an iterative process of the following two steps:

• The E-step:

Find the expectation of the log-likelihood function of the model parameter θ, to be maximized in the following M-step.

Find the likelihood function of θ based on the observed dataset X = [x_{1}, ⋯, x_{N}] (all samples assumed to be i.i.d.):

L(θ|X, z) = p(X, z|θ) = ∏_{n=1}^{N} p(x_{n}, z|θ) = ∏_{n=1}^{N} p(x_{n}|z, θ) p(z|θ) (10.51)

and the log-likelihood:

log L(θ|X, z) = ∑_{n=1}^{N} (log p(x_{n}|z, θ) + log p(z))  (10.52)

The second term can be dropped as p(z) = N (0, I) is independent of the model parameter θ and therefore irrelevant to the maximization of the log-likelihood with respect to θ.

We then find the expectation of the log-likelihood function, denoted by Q, with respect to the latent variable z based on Eq. (10.47):

Q = E_{z|x} [log L(θ|X, z)] = ∑_{n=1}^{N} E_{z|x_{n}} [log p(x_{n}|z, θ)]

= ∑_{n=1}^{N} E_{z|x_{n}} log [\frac{1}{(2π)^{d/2}|Σ_{x|z}|^{1/2}} exp (−\frac{1}{2}(x_{n} − m_{x|z})^{T} Σ_{x|z}^{−1}(x_{n} − m_{x|z}))]

= ∑_{n=1}^{N} E_{z|x_{n}} [−\frac{d}{2} log(2π) − \frac{1}{2} log |Ψ| − \frac{1}{2}(x_{n} − Wz)^{T} Ψ^{−1}(x_{n} − Wz)]

= −\frac{dN}{2} log(2π) − \frac{N}{2} log |Ψ| − \frac{1}{2} ∑_{n=1}^{N} E_{z|x_{n}} [(x_{n} − Wz)^{T} Ψ^{−1}(x_{n} − Wz)]

= C − \frac{N}{2} log |Ψ| − \frac{1}{2} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} Ψ^{−1}x_{n} − 2x_{n}^{T} Ψ^{−1}Wz + z^{T} W^{T} Ψ^{−1}Wz] (10.53)

where we have used the fact that

x_{n}^{T} Ψ^{−1}(Wz) = (Wz)^{T} Ψ^{−1}x_{n}  (10.54)

as both Ψ^{−1} and Ψ are diagonal and therefore also symmetric. Here the constant C = −dN log(2π)/2 can be dropped.

• The M-step:

Find the optimal model parameter θ = {W, Ψ} that maximizes the expectation of the log-likelihood Q obtained in the E-step.

We set to zero the derivative of Q with respective each of the two parameters in θ = {W, Ψ} and solve the resulting equations.

– Find W:

\frac{∂Q}{∂W} = \frac{∂}{∂W} ∑_{n=1}^{N} E_{z|x_{n}} [−x_{n}^{T} Ψ^{−1}x_{n} + 2x_{n}^{T} Ψ^{−1}Wz − z^{T} W^{T} Ψ^{−1}Wz]

= ∑_{n=1}^{N} E_{z|x_{n}} [\frac{∂}{∂W} (2x_{n}^{T} Ψ^{−1}Wz − z^{T} W^{T} Ψ^{−1}Wz)]

= ∑_{n=1}^{N} E_{z|x_{n}} [2Ψ^{−1}x_{n}z^{T} − 2Ψ^{−1}Wzz^{T} ]

= 2Ψ^{−1} ∑_{n=1}^{N} (x_{n}E_{z|x_{n}}[z^{T}] − WE_{z|x_{n}}[zz^{T}]) = 0  (10.55)

Solving for W we get

W = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z^{T}]) (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}])^{−1}  (10.56)

where E_{z|x_{n}}[z] and E_{z|x_{n}}[zz^{T}] can be found in Eq. (10.48):

\{ \begin{matrix}E_{z|x_{n}}[z] & = & m_{z|x_{n}} = Bx_{n} \\ E_{z|x_{n}}[zz^{T}] & = & Σ_{z|x_{n}} + m_{z|x_{n}}m_{z|x_{n}}^{T} = I − BW + Bx_{n}x_{n}^{T} B^{T} \end{matrix}

(10.57)

The second equation is due to the fact that Σ_{z} = E[zz^{T}] − m_{z}m_{z}^{T}. Here E_{z|x_{n}}[z] can be considered as the estimation of the z, while E_{z|x_{n}}[zz^{T}] the uncertainty of the estimation.

– Find Ψ:

\frac{∂Q}{∂Ψ^{−1}} = \frac{∂}{∂Ψ^{−1}} [N log |Ψ| + ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} Ψ^{−1}x_{n} − 2x_{n}^{T} Ψ^{−1}Wz + z^{T} W^{T} Ψ^{−1}Wz]]

= −NΨ + ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}x_{n}^{T} − 2x_{n}z^{T} W^{T} + W(zz^{T})W^{T} ]

= −NΨ + ∑_{n=1}^{N} x_{n}x_{n}^{T} − 2 ∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} W^{T} + W ∑_{n=1}^{N} E_{z|x}[zz^{T}]W^{T} = 0(10.58)

Solving for Ψ we get

Ψ = \frac{1}{N} diag (∑_{n=1}^{N} x_{n}x_{n}^{T} − 2 (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} ) W^{T} + W (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}]) W^{T} )

(10.59)

where diag(A) is the operation that sets all off-diagonal elements of matrix A to zero, so that the resulting matrix is indeed diagonal as what Ψ should be. We further replace W in front of the last term above by that in Eq. (10.56) and get

Ψ = \frac{1}{N} diag (∑_{n=1}^{N} x_{n}x_{n}^{T} − ∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} W^{T} )  (10.60)

In summary, here are the steps of the EM method:

1. Initialize parameters θ_{old} = {W_{old}, Ψ_{old}};

2. E-step:

Find E_{z|x}(z) and E_{z|x}(zz^{T}) in Eq. (10.57) based on m_{z|x_{n}} and Σ_{z|x_{n}} in Eq. (10.48), which in turn is based on θ_{old} = {W_{old}, Ψ_{old}};

3. M-step:

Find θ_{new} = {W_{new}, Ψ_{new}} in Eqs. (10.56) and (10.60), based on E_{z|x}(z) and E_{z|x}(zz^{T});

4. Terminate if convergence cretierion is satisfied, otherwise replace θ_{old} by θ_{new} and return to the E-step.

As the E-step and M-step of the EM algorithm for FA are interdependent on each other, they need to be carried out iteratively based on cetain inital guess of the parameters in θ, as show in the steps below:

10.4 Probabilistic PCA

The method of probabilistic PCA (PPCA) can be considered as a special case of factor analysis based on the model x = Wz + e, when the random noise e is assumed to have an isotropic normal distributions with covariance matrix Ψ = εI. More specially, if we further let ε → 0 approach zero, the iteration of the EM algorithm for the PPCA becomes extremely simple, as we will see below.

Same as in Eqs. (10.36) and (10.38) for FA, here we also have the normal distribution of both z and e:

p(z) = N (0, I),  p(e) = N (0, εI)  (10.61)

and the normal distribution of the observed data x = Wz + e, same as in Eq. (10.42):

p(x) = N (0, WW^{T} + εI)  (10.62)

The goal here is to estimate ε and W as the model parameter, based on the observed data dataset X = [x_{1}, ⋯, x_{N}].

Now Eqs. (10.48) through (10.50) are rewritten as:

\{ \begin{matrix}m_{z|x_{n}} = E_{z|x_{n}}(z) = Bx_{n} \\ Σ_{z|x_{n}} = Cov_{z|x_{n}}(z) = I − BW \end{matrix}  (10.63)
B = W^{T} (WW^{T} + εI)^{−1}  (10.64)
(WW^{T} + εI)^{−1} = ε^{−1}I − ε^{−1}W(εI + W^{T} W)^{−1}W^{T}  (10.65)

The two steps of the EM algorithm become:

• E-step: Get the expectation of the log-likelihood by replacing Ψ in Eq. (10.53) by εI:

Q = −\frac{dN}{2} log ε − \frac{1}{2ε} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} x_{n} − 2x_{n}^{T} Wz + z^{T} W^{T} Wz] (10.66)

• M-step: Find the optimal model parameters W and ε that maximize the expectation of the log-likelihood Q above, by setting the derivative of Q with respect to the parameters to zero and solve the resulting equations. For W, the result is the same as Eq. (10.56):

W_{new} = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z^{T}]) (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}])^{−1}  (10.67)

where

\{ \begin{matrix}E_{z|x_{n}}[z] & = & m_{z|x_{n}} = Bx_{n} \\ E_{z|x_{n}}[zz^{T}] & = & Σ_{z|x_{n}} + m_{z|x_{n}}m_{z|x_{n}}^{T} \end{matrix}  (10.68)

To find ε, we solve the equation

\frac{∂Q}{∂ε} = −\frac{dN}{2ε} + \frac{1}{2ε^{2}} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} x_{n} − 2x_{n}^{T} Wz + z^{T} W^{T} Wz] = 0 (10.69)

and get

ε = \frac{1}{dN} ∑_{n=1}^{N} [x_{n}^{T} x_{n} − 2x_{n}^{T} WE_{z|x_{n}}(z) + tr(W^{T} E_{z|x_{n}}(zz^{T})W)]

= \frac{1}{dN} ∑_{n=1}^{N} [||x_{n}||^{2} − 2E_{z|x_{n}}(z^{T})W^{T} x_{n} + tr(E_{z|x_{n}}(zz^{T})W^{T} W)] (10.70)

where we have used the fact that a^{T} b = tr(ab^{T}).

In summary, here are the steps in the EM algorithm for PPCA:

1. Initialize parameters θ_{old} = {W_{old}, ε_{old}};

2. E-step: Based on θ_{old}, find m_{z|x_{n}} and Σ_{z|x_{n}} in Eq. (10.63), and E_{z|x_{n}}(z) and E_{z|x_{n}}(zz^{T}) in Eq. (10.68);

3. M-step: Find θ_{new} = {W_{new}, ε_{new}} by evaluating Eqs. (10.67) and (10.70);

4. Terminate if convergence cretierion is satisfied, otherwise replace θ_{old} by θ_{new} and return to step 2.

Specially, if we further assume ε → 0 and the latent variables in z are deterministic, i.e., E_{z|x}(z) = z. At the limit where ε = 0, x is simply a linear combination of the latent variables in z:

x = Wz + εI ⟹^{ε→0} Wz = [w_{1}, ⋯, w_{d′}] [ z_{1} ; ⋮ ; z_{d′} ] = ∑_{i=1}^{d′} z_{i}w_{i}  (10.71)

Comparing this expression of x with Eq. (9.44)

x = V′y = ∑_{i=1}^{d′} y_{i}v_{i}  (10.72)

we see that the column vectors in W = [w_{1}, ⋯, w_{d′}] for the PPCA are similar to the column vectors of V , the eigenvector matrix of Σ_{x}, in the linear PCA, as both of them can be viewed as the principal directions, an alternative set of basis vectors that also span the same d-dimensional feature space in which x resides. When x is expressed as a linear combination of these basis vectors, it can be approximated by only d′ < d of the d such basis vectors weighted by the d′ principal components in y = [y_{1}, ⋯, y_{d′}]^{T} in linear PCA, or the d′ factors in z = [z_{1}, ⋯, z_{d′}]^{T} in PPCA.

Here are the two EM steps for the PPCA:

• E-step: If ε → 0, Eq. (10.63) can be written as:

m_{z|x_{n}} = E_{z|x_{n}}(z) = z_{n} = Bx_{n} = W^{T} (WW^{T} + εI)^{−1}x_{n}
⟹^{ε→0} W^{T} (WW^{T})^{−1}x_{n} (n = 1, ⋯, N)  (10.73)

All N such equations can be written in matrix form as

Z = [z_{1}, ⋯, z_{n}] = W^{T} (WW^{T})^{−1}[x_{1}, ⋯, x_{N}] = W^{T} (WW^{T})^{−1}X

(10.74)

As W is a d × d′ matrix (d′ < d), the rank of the d × d matrix WW^{T} is at most d′, i.e., the inverse of WW^{T} does not exist and we cannot find z_{n} = W^{T} (WW^{T})^{−1}x_{n}. In fact the transformation matrix W^{T} (WW^{T})^{−1} is the right psedo inverse (Section A.15) of the d × d′ matrix W, which exists only if d′ ≥ d.

However, we note that the model x = Wz is an over-constraned linear system of d equations but only d′ < d variables in z = [z_{1}, ⋯, z_{d′}]^{T}, and the least-squares solution that minimizes the squared error ||x−Wz||^{2} can be found by the left pseudo inverse (Section A.15) of W:

z_{n} = (W^{T} W)^{−1}W^{T} x_{n} = W^{−}x_{n} (n = 1, ⋯, N)  (10.75)

and we can get the latent variables in Z = [z_{1}, ⋯, z_{N}] given all N data points in X = [x_{1}, ⋯, x_{N}]:

Z = (W^{T} W)^{−1}W^{T} X = W^{−}X  (10.76)

which is based on the old value of W previously estimated.

• M-step: Now Eq. (10.67) can be written as:

W = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}(z^{T})) (∑_{n=1}^{N} E_{z|x_{n}}(zz^{T}))^{−1}

= [x_{1}, ⋯, x_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] ([z_{1}, ⋯, z_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ])^{−1} = XZ^{T} (ZZ^{T})^{−1} = XZ^{−} (10.77)

by which W is uptated based on Z found previously in the E-step. We note that Z^{T} (ZZ^{T})^{−1} is the right pseudo inverse of the d × N matrix Z, which exists if d ≤ N.

Again, as the E-step and M-steps are interdependent on each other, they need to be carried out iteratively:

E-step:  Z = (W^{T} W)^{−1}W^{T} X = W^{−}X
M-Step:  W = XZ^{T} (ZZ^{T})^{−1} = XZ^{−}  (10.78)

We note that given the dataset X, the FA model x = Wz can be expressed as an over-determined matrix equation

X_{d×N} = [x_{1}, ⋯, x_{N}] = W_{d×d′}[z_{1}, ⋯, z_{N}] = W_{d×d′}Z_{d′×N}  (10.79)

or, taking transpose of the equation, we get

X_{N×d}^{T} = Z_{N×d′}^{T}W_{d′×d}^{T}  (10.80)

Given X, this equation can be interpreted in two ways:

• If W is available, this is a system of d equations but d′ < d unknowns in each of the N columns of Z, which is solved as in the E-step above;

• If Z is available, this is a system of N equations but d′ < N unknowns in each of the d rows of W, which is solved as in the M-step above.

In either step, the pseudo inverse method can be used to minimize the squared error ||X − WZ||^{2}.

When compared to the regular PCA method that finds all d eigenvalues and the corresponding eigenvectors all at once by solving the eigenequation Σ_{x}w = λw, the PPCA method only finds the d′ column vectors of matrix W as the basis vectors, as the basis vectors that span a subspace with much reduced dimensionality but containing the most essential information in the data. However, unlike the PCA, the basis vectors in PPCA are not necessarily orthogonal to each other.

The implementation of the PPCA is extremely simple, as shown in the Matlab code below. Based on some initialized W, the EM iteration is composed of the E-step and M-step:

    while er>tol
        Z=inv(W'*W)*W'*X;       % find Z given W in E-step
        Wnew=X*Z'*inv(Z*Z');    % find W given Z in M-step
        er=norm(Wnew-W);        % the LS error
        W=Wnew;
    end

Example 1:

The figure below shows the first few iterations of the PPCA based on the EM method presented above, for a set of N = 1000 data points in X = [x_{1}, ⋯, x_{N}] the d = 2 dimensional space. The stright line in the plots indicates the direction of w_{1}, which is initially set along a random direction, but quickly approaches the direction corresponding to the principal component obtained by the PCA method, along which the data points are most widely spread.

Example 2:

The figure below shows the first few iterations of the PPCA EM algorithm for a set of data points in d = 3 dimensional space. The stright lines in the plots indicates the direction of the principal components when d′ = 2.

Example 3:

The figure below shows the visualization of the dataset for the hand-written numbers from 0 to 9, containing 2240 data points in a 256-D space, but now linearly mapped to a 3-D space spanned by d′ = 3 factors found by the PPCA method.

10.5 Classical Multidimensional Scaling

The goal of multidimensional scaling (MDS) is to reduce the dimensionality of the dataset representing a set of N objects of interest, each described by a set of d features and represented as a vector x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional space, while the pairwise similarity relationship between any two of these objects are preserved. MDS can be used to visualize datasets in a high dimensional space. Specially, here we consider the classical MDS (cMDS), also known as principal coordinates analysis (PCoA), as one of the methods in MDS.

Specifically, we represent the pairwise similarity of a set of N objects X = [x_{1}, ⋯, x_{N}] by the N × N similarity matrix D_{x} = [ d_{ij}^{x} ], of which the ij-th component d_{ij}^{x} is certain measurement of the similarity between x_{i} and x_{j}. The goal is to map the dataset X in the d-dimensional space into Y = [y_{1}, ⋯, y_{N}] in a lower d’-dimensional space, in which the similarity matrix D_{x} is optimally approximated by D_{y} = [ d_{ij}^{y}]. When d′ ≤ 3, these data points in Y can be visualized. MDS can be considered as a method for dimension reduction. In some cases only the pairwise similarities d_{ij}^{x} (i, j = 1, ⋯, N) are given, while the N objects in X are not explicitly given, and the dimensionality d may be unknown.

We assume the given pairwise similarity is the Euclidean distance d_{ij} = ||x_{i} − x_{j}||_{2} between x_{i} and x_{j}:

d_{ij}^{2} = ||x_{i} − x_{j}||^{2} = (x_{i} − x_{j})^{T} (x_{i} − x_{j}) = x_{i}^{T} x_{i} − 2x_{i}^{T} x_{j} + x_{j}^{T} x_{j} (10.81)

and get

D_{x}^{2} = [ d_{ij}^{2}  ]_{N×N} = [ x_{i}^{T} x_{i} − 2x_{i}^{T} x_{j} + x_{j}^{T} x_{j}  ]_{N×N} = X_{r}−2X^{T} X+X_{c}

(10.82)

where

X_{r} = [ \begin{matrix}||x_{1}||^{2} & ⋯ & ||x_{1}||^{2} \\ ⋮ & ... & ⋮ \\ ||x_{N}||^{2} & ⋯ & ||x_{N}||^{2} \end{matrix} ] = X_{c}^{T}

X^{T} X = [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] [x_{1}, ⋯, x_{N}] = [ \begin{matrix}x_{1}^{T} x_{1} & ⋯ & x_{1}^{T} x_{N} \\ ⋮ & ... & ⋮ \\ x_{N}^{T}x_{1} & ⋯ & x_{N}^{T}x_{N} \end{matrix} ] (10.83)

Our goal is to find Y = [y_{1}, ⋯, y_{N}] in a lower dimensional space so that its pairwise similarity matrix D_{y} matches D_{x} optimally, in the sense that the following objective function is minimized:

J(Y) = ||D_{y}^{2} − D_{x}^{2}||  (10.84)

We note that such a solution Y is not unique, as any translated version of Y has the same pairwise similarity matrix and is also a solution. We therefore seek to find a unique solution Y in which all data points are centralized, i.e, the mean of all points in Y is zero:

ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n} = 0  (10.85)

To do so, we introduce an N × N symmetric centering matrix C = I − 1/N = C^{T}, where1 = ee^{T} is an N ×N matrix with all components equal 1. Premultiplying C to a column vector a = [a_{1}, ⋯, a_{N}]^{T} removes the mean ā = ∑_{i=1}^{N} a_{i}/N of a from each of its component:

Ca = Ia−\frac{1}{N} 1a = [ \begin{matrix}1 & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 1 \end{matrix} ] [ a_{1} ; ⋮ ; a_{N} ]−\frac{1}{N} [ \begin{matrix}1 & ⋯ & 1 \\ ⋮ & ... & ⋮ \\ 1 & ⋯ & 1 \end{matrix} ] [ a_{1} ; ⋮ ; a_{N} ] = [ a_{1} − ā ; ⋮ ; a_{N} − ā ]

(10.86)

Taking transpose of the equation above, we get

(Ca)^{T} = a^{T} C = [a_{1} − ā, ⋯, a_{N} − ā]  (10.87)

i.e., postmultiplying C to a row vector a^{T} = [a_{1}, ⋯, a_{N}] removes the mean ā from each of its component.

We can now postmultiply C to X to remove the mean of each row of X:

XC = [x_{1}, ⋯, x_{N}]C = [x̄_{1}, ⋯, x̄_{N}] = X̄  (10.88)

i.e., the nth column becomes:

x̄_{n} = x_{n} − x̄, where x̄ = \frac{1}{N} ∑_{n=1}^{N} x_{n}  (10.89)

and premultiply C to X^{T} to remove the mean of each column of X^{T}:

CX^{T} = X̄^{T}  (10.90)

We further apply double centering to D_{x}^{2} by pre and post-multiplying C and get:

CD_{x}^{2}C = C (X_{r} − 2X^{T} X + X_{c}) C = CX_{r}C − 2C (X^{T} X) C + CX_{c}C
= −2C (X^{T} X) C = −2 (XC)^{T} (XC) = −2X̄^{T} X̄

Note that the first term CX_{r}C = C(X_{r}C) = C0 = 0, as all components of each row of X_{r} are the same as their mean, and removing the mean results in a zero row vector, X_{r}C = 0; and the last term CX_{c}C = (CX_{c})C = 0C = 0, as all components of each column of X_{c} are the same as their mean, and removing the mean results in a zero column vector.

Now the similarity matrix D_{x}^{2} of all centralized data points in X̄ with zoero mean can be represented by the Gram matrix of X̄:

Gx̄ = X̄^{T} X̄ = −\frac{1}{2}CD_{x}^{2}C  (10.91)

We further assume all data points in Y are also centeralized with zero mean and their corresponding similarity matrix is also represented by Gȳ, then the objective function can be expressed as:

J(Y) = ||Gȳ − Gx̄|| = ||Ȳ^{T} Ȳ − Gx̄||  (10.92)

To find Ȳ that minimizes J(Y), we first consider the equation Ȳ^{T} Ȳ − Gx̄ = 0 so that J(Y) = 0. In other words, we desire to express Gx̄ as the inner product of some matrix with itself. To do so, we carry out eigenvalue decomposition of G_{x} to find its eigenvalue matrix Λ and the orthogonal eigenvector matrix V satisfying G_{x}V = VΛ, i.e.,

Gx̄ = VΛV^{−1} = VΛV^{T} = [v_{1}, ⋯, v_{N}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{N} \end{matrix} ] [ v_{1}^{T} ; ⋮ ; v_{N}^{T} ]

= (VΛ^{1/2}) (Λ^{1/2}V^{T} ) = (Λ^{1/2}V^{T} )^{T} (Λ^{1/2}V^{T} )

Note that as Gx̄ is symmetric, the eigenvalues are real and the eigenvectors are orthogonal, i.e., V^{T} = V^{1} is is orthogonal matrix. Now we can find the desired Y as:

Y = Λ^{1/2}V^{T} = [ \begin{matrix}\sqrt{λ_{1}} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & \sqrt{λ_{N}} \end{matrix} ] [ v_{1}^{T} ; ⋮ ; v_{N}^{T} ] = [ \sqrt{λ_{1}}v_{1}^{T} ; ⋮ ; \sqrt{λ_{N}}v_{N}^{T} ] (10.93)

Each column y_{i} in Y = [y_{1}, ⋯, y_{N}] is an N-D vector. If all N dimensions of these y_{n} are used, Y^{T} Y = G_{x} and J(Y = 0. To reduce the dimensionality to d′ < d, we construct Y′ by the first d′ rows of Y corresponding to the d′ greatest eigenvalues of λ_{1} ≥ ⋯ ≥ λ_{d′} ≥ ⋯ ≥ λ_{N} and their corresponding eigenvectors v_{1}, ⋯, v_{d′}. The error is:

J(Y′) = ||Y^{′T} Y′ − Y^{T} Y|| = || ∑_{i=1}^{d′} λ_{i}v_{i}v^{T} − ∑_{i=1}^{N} λ_{i}v_{i}v^{T} || = || ∑_{i=d′}^{N} λ_{i}v_{i}v^{T} ||

(10.94)

In summary, here is the PCoA algorithm:

• Given the pairwise similarity, an N × N matrix D_{x} = [ d_{ij} ], construct the squared proximity matrix D_{x}^{2} = [ d_{ij}^{2} ];

• Apply double centering to get G_{x} = −CD_{x}^{2}C/2; • Find the d′ greatest eigenvalues λ_{1}, ⋯, λ_{d′} of G_{x}, and the corresponding

eigenvectors v_{1}, ⋯, v_{d′};

• Get the map

Y_{d′×N} = Λ_{d′}^{1/2}V_{d′} = [ \sqrt{λ_{1}}v_{1}^{T} ; ⋮ ; \sqrt{λ_{d′}}v_{d′}^{T} ]  (10.95)

The Matlab functions for the PCA and PCoA are listed below:

function Y=PCA(X)
[V D]=eig(cov(X'));         % solve eigenequation for covariance of data
[d idx]=sort(diag(D),'descend');    % sort eigenvalues in descend order
V=V(:,idx);                         % reorder eigenvectors
Y=V'*X;                             % carry out KLT
end

function Y=PCoA(X)
N=size(X,2);
D=zeros(N);                         % pairwise similarity matrix
C=eye(N)-ones(N)/N;                 % centering matrix
for i=1:N
    for j=1:N
        D(i,j)=(norm(X(:,i)-X(:,j)))^2;
    end
end
B=-C*D*C/2;
[V D]=eig(B);                       % solve eigenequation for matrix B
[d idx]=sort(diag(D),'descend');    % sort eigenvalues in descend order
V=V(:,idx);                         % reorder eigenvectors
Y=sqrt(D)*V';                       % carry out PCoA
end

Example 1: In the figure below, the top panel shows the locations of some major cities

in North America, and the bottom panel shows the reconstructed city locations based on the pairwise distance of these cities. Note that the reconstructed map is a rotated and centralized version of the original map.

Example 2: Both PCA and PCoA are applied to the Iris dataset of 150 4-D data points,

and the figure below shows the data points in 2-D and 3-D spaces corresponding to the first 2 and 3 eigenvalues.

Example 3: Both PCA and PCoA are applied to the dataset of handwritten numbers of 2240 256-D data points, and the figure below shows the data points in 2-D and 3-D spaces corresponding to the first 2 and 3 eigenvalues.

10.6 t-Distributed Stochastic Neighbor Embedding (t-SNE)

It is impossible to reduce the dimensionality of a given dataset which is intrinsically high-dimensional (high-D), while still preserving all the pairwise distances in the resulting low-dimensional (low-D) space, compromise will have to be made to sacrifice certain aspects of the dataset when the dimensionality is reduced.

The PCA method as a linear, dimensionality reduction algorithm, is unable to interpret complex nonlinear relationship between features. tance

preserving method, the PCA emphasizes the importance of the PCA dimensions with great variances, but neglects small distance variations. It places distant and dissimilar data points far apart in the low-D space, while ignoring the importance of similar datapoints close together in the high-D, which need to be also placed close together in the low-D space. Therefore the PCA method tends to preserve large scale global structure of the dataset, while ignoring the small scale local structure.

But in some cases the local structures are of equal or even more importance than the global ones, such as the typical example of “Swiss roll”.

Also, when the dimensionality is much reduced from a high-D space to a low-D space, the crowding problem occurs. In a high-D space, a data point may have a large number of similar neighbors of roughly equal distance; however, in a low-D space, the number of equal-distance neighbors is significantly reduced (8 in a 3-D space, 4 in a 2-D space, 2 in a 1-D space). Thus the space available for the neighbors of a point is much reduced. As the result, when a large number of neighbors in the high-D are mapped to a low-D space, they are forced to spread out, some are getting closer to those that are farther away in the original space, thereby reducing the gaps between potential clusters of similar points.

In contrast, the t-SNE method is a nonlinear method that is based on probability distributions of the data points being neighbors, and it attempts to preserve the structure at all scales, but emphasizing more on the small scale structures, by mapping nearby points in high-D space to nearby points in low-D space.

The similarity between any two distinct points x_{i} and x_{j} in the high-D space is defined as a probability based on the Gaussian kernel:

p_{ij} = \frac{exp(−||x_{i} − x_{j}||^{2}/2σ^{2})}{∑_{k} ∑_{l≠k} exp(−||x_{k} − x_{l}||^{2}/2σ^{2})},  ∑_{i} ∑_{j} p_{i/j} = 1  (10.96)

However, if x_{i} = x_{j}, p_{ij} = 0 is defined to be zero, as it never needed in the algorithm. Note that p_{ij} is a probability measurement as it is normalized and add up to 1.

If x_{j} is close to x_{j}, ||x_{i}−x_{j}|| is small and p_{ij} is large, indicating x_{j} is similar to x_{i}; but if x_{j} is far away from x_{j}, ||x_{i} −x_{j}|| is large and p_{ij} is small, indicating x_{j} is dissimilar to x_{i}. Also, as ∑_{i} ∑_{j} p_{ij} = 1, p_{ij} can be considered as a probability that any two points x_{j} and x_{i} are neighbors in the feature space and are therefore similar to each other.

Similarly, we could also define the similarity q_{ij} in the low-D space also based on the Gaussian kernel. However, for reason explained below, we prefer to define the similarity q_{in} in the low-D space based on Student’s t-distribution of degree 1, instead of Gaussian distribution:

q_{ij} = \frac{(1 + ||y_{i} − y_{j}||^{2})^{−1}}{∑_{k} ∑_{l≠k}(1 + ||y_{k} − y_{l}||^{2})^{−1}}  (10.97)

As shown in the figure below, when compared with the Gaussian distribution, the t-distribution has a lower peak in the center but higher tails on the sides. It is because of this property, the t-distribution is used to address the crowding problem. Let x_{i} be the point of interest (black, in the center of the Gaussian kernel), then another point x_{j} (red) is mapped to a different position, which is closer to x_{i} if they are initially close (similar), or farther away from each other if they are initially far apart (dissimilar).

How well the points y_{i} in the low-D space represent the similarity relationship among the points x_{i} in the original high-D space can be measured by the discrepancy bwtween the two sets of probabilities, measured by the cost function, defined as the Kullback-Leibler divergence (Section 3.2.2):

C = D_{KL}(P ||Q) = ∑_{i} ∑_{j} p_{ij} log \frac{p_{ij}}{q_{ij}} = ∑_{i} ∑_{j} (p_{ij} log p_{ij} − p_{ij} log q_{ij}) (10.98)

We desire to find the optimal map points y_{i} in the low-D space that minimize the discrepancy C = D_{KL}(P ||Q).

The KL-divergence is not symmetric as D_{KL}(P ||Q) ≠ D_{KL}(Q||P), therefore it can not be viewed as a distance between the two sets of probabilities. This asymmetry is actually a desired feature. Consider the following two cases:

• If p_{ij} is large but q_{ij} is small, the cost C is very large, i.e., and such a mapping is heavily penalized. As the result, similar points that are close to each other in the high-D space tend to be mapped to points in the low-D space that are also close to each other.

• If p_{ij} is small but q_{ij} is large, the cost C is small, i.e., such a mapping is not heavily penalized. As the result, dissimilar points that are far apart in the high-D space are allowed to be mapped to points in the low-D space that are closer to each other.

We therefore see that minimizing the KL-divergence has the effect of preserving the local structure of the dataset, but not necessarily the global structure.

Before proceeding to consider the minimization of the cost function C, we still need to resolve two additional issues:

• One potential problem with p_{ij} defined above is that when a point x_{i} is an outlier with large distances ||x_{i}−x_{j}|| to every other point x_{j} in the dataset, p_{ij} is small and insignificant in the cost function C, and the location of y_{i} in the low-D space can not be well determined. To resolve this issue, we redefine p_{ij} as the average of two conditioinal probablities:

p_{ij} = \frac{p_{i/j} + p_{j/i}}{2n}  (10.99)

where n is the total number of data points and

p_{i/j} = \frac{exp(−||x_{i} − x_{j}||^{2}/2σ_{i}^{2})}{∑_{k≠i} exp(−||x_{i} − x_{k}||^{2}/2σ_{i}^{2})},  ∑_{i} p_{j/i} = 1  (10.100)

is the conditional probability that any point x_{j} is a neighboring point similar to a given point x_{i}. Note that p_{j/i} is properly normalized and add up to 1:

∑_{j=1}^{n} ∑_{i=1}^{n} p_{i/j} = \frac{1}{2n} ∑_{i=1}^{n} (∑_{j=1}^{n} p_{j/i})+\frac{1}{2n} ∑_{j=1}^{n} (∑_{i=1}^{n} p_{i/j}) = \frac{1}{2n} ∑_{i=1}^{n} 1+\frac{1}{2n} ∑_{j=1}^{n} 1 = \frac{1}{2n}(n+n) = 1

(10.101)

i.e., it is a probability measurement.

• Whether two points are close (therefore similar) to each other needs to be measured relatively, independent of the local data density, which may vary in different local regions of the feature space. We can therefore use different variance σ_{i} for p_{j/i} depending on the local data density in the neighborhood of x_{i}. Specifically, we choose σ_{i} in such a way that the perplexity (Section 3.2.1) 2^{H(p_{i})} is the same for all data points, i.e., the entropy H(p_{i}) = − ∑_{j} p_{j/i} log_{2} p_{j/i} representing the uncertainty is the same, independent of the local data density. As shown in Section 3.2.1, the entropy of a Gaussian p(x, μ, σ) is log(2πeσ^{2})/2, a monotonic function of its variance σ.

We can now find the optimal map points y_{i} that minimize the cost function C = D_{KL}(P ||Q), by gradient descent method.

To find the gradient of ∂C/∂y_{i}, we first introduce a set of intermediate variables

d_{kl} = ||y_{k}−y_{l}|| = [(y_{k} − y_{l})^{T} (y_{k} − y_{l})]^{1/2} (k, l = 1, ⋯, n), Z = ∑_{k} ∑_{l≠k}(1+d_{kl}^{2})^{−1}

(10.102)

so that

q_{ij} = \frac{(1 + ||y_{i} − y_{j}||^{2})^{−1}}{∑_{k} ∑_{l≠k}(1 + ||y_{k} − y_{l}||^{2})^{−1}} = \frac{(1 + d_{ij}^{2})^{−1}}{∑_{k} ∑_{l≠k}(1 + d_{kl}^{2})^{−1}} = \frac{(1 + d_{ij}^{2})^{−1}}{Z}

(10.103)

Note that

\frac{∂Z}{∂d_{ij}} = \frac{∂}{∂d_{ij}} [∑_{k} ∑_{l≠k}(1 + d_{kl}^{2})^{−1}] = −2d_{ij}(1 + d_{ij}^{2})^{−2}  (10.104)

We now find ∂C/∂y_{i} based on chain-rule. Note that when y_{i} changes, only d_{ij} and d_{ji} (for all j) change, we therefore have

\frac{∂C}{∂y_{i}} = ∑_{j} \frac{∂C}{∂d_{ij}} \frac{∂d_{ij}}{∂y_{i}} + ∑_{j} \frac{∂C}{∂d_{ji}} \frac{∂d_{ji}}{∂y_{i}}  (10.105)

But as

\frac{∂d_{ij}}{∂y_{i}} = \frac{∂d_{ji}}{∂y_{i}} = \frac{∂}{∂y_{i}} [(y_{i} − y_{j})^{T} (y_{i} − y_{j})]^{1/2} = \frac{1}{2} [(y_{i} − y_{j})^{T} (y_{i} − y_{j})]^{−1/2} 2(y_{i}−y_{j}) = \frac{y_{i} − y_{j}}{d_{ij}}

(10.106)

we have

\frac{∂C}{∂y_{i}} = ∑_{j} (\frac{∂C}{∂d_{ij}} + \frac{∂C}{∂d_{ji}} ) \frac{y_{i} − y_{j}}{d_{ij}} = 2 ∑_{j} \frac{∂C}{∂d_{ij}} \frac{y_{i} − y_{j}}{d_{ij}}  (10.107)

We next find ∂C/∂d_{ij}. Note that only q_{kl} may be a function of d_{ij}, while all p_{kl} are constants, we therefore have

\frac{∂C}{∂d_{ij}} = \frac{∂}{∂d_{ij}} [∑_{k} ∑_{l≠k} (p_{kl} log p_{kl} − p_{kl} log q_{kl})] = − ∑_{k} ∑_{l≠k} p_{kl} \frac{∂}{∂d_{ij}} log q_{kl}

= − ∑_{k} ∑_{l≠k} p_{kl} \frac{∂}{∂d_{ij}} (log q_{kl}Z − log Z) = ∑_{k} ∑_{l≠k} p_{kl} (−\frac{1}{q_{kl}Z} \frac{∂}{∂d_{ij}} (1 + d_{kl}^{2})^{−1} + \frac{1}{Z} \frac{∂Z}{∂d_{ij}} )

= p_{ij} (\frac{1}{q_{ij}Z} (1 + d_{ij}^{2})^{−2} 2d_{ij}) − (∑_{k} ∑_{l≠k} p_{kl}) \frac{1}{Z} (1 + d_{ij}^{2})^{−2} 2d_{ij}

= 2d_{ij} [\frac{p_{ij}}{q_{ij}} \frac{(1 + d_{ij}^{2})^{−2}}{Z} − \frac{(1 + d_{ij}^{2})^{−2}}{Z} ]  (10.108)

The second to the last equality is due to ∂(1 + d_{kl}^{2})^{−1}/∂d_{ij} = 0 when i ≠ k or j ≠ l, and the last equality is due to ∑_{k} ∑_{l≠k} p_{kl} = 1. But as

\frac{(1 + d_{ij}^{2})^{−2}}{Z} = q_{ij}(1 + d_{ij}^{2})^{−1}  (10.109)

The above becomes

\frac{∂C}{∂d_{ij}} = 2d_{ij} (\frac{p_{ij}}{q_{ij}} − 1) \frac{(1 + d_{ij}^{2})^{−2}}{Z} = 2d_{ij}(p_{ij} − q_{ij})(1 + d_{ij}^{2})^{−1}  (10.110)

We can now finally get the gradient vector of the cost function C with respect to each of the map points y_{i}:

\frac{∂C}{∂y_{i}} = 2 ∑_{j} \frac{∂C}{∂d_{ij}} \frac{y_{i} − y_{j}}{d_{ij}} = 4(p_{ij} − q_{ij})(1 + d_{ij}^{2})^{−1}(y_{i} − y_{j})
= 4 ∑_{j}(p_{ij} − q_{ij}) (1 + ||y_{i} − y_{j}||^{2})^{−1} (y_{i} − y_{j})  (10.111)

10.6 t-Distributed Stochastic Neighbor Embedding (t-SNE) 261 Given the gradient vector, we can iteratively approach the optimal data points

in the low-D space that minimize the discrepancy C = D_{KL}(P ||Q):

y_{n+1} = y_{n} + δ \frac{∂C}{δy} + α_{n}(y_{n} − y_{n−1})  (10.112)

where δ is the step size (learning rate) and the second term is momentum term.

11 Independent Component Analysis

Independent Component Analysis

11.1 Independence and Non-Gaussianity

The goal of independent component analysis (ICA) is to reconstruct a set of latent source signals denoted by s = [s_{1}, ⋯, s_{m}]^{T} and assumed to be independent of each other, thereby the term independent components, from the observed variables x = [x_{1}, ⋯, x_{n}]^{T} as the linear combinations of the source signals,

x = As = ∑_{i=1}^{m} a_{i}s_{i}  (11.1)

where A = [a_{1}, ⋯, a_{m}] is an n×m mixing matrix of unknown values. We assume n ≥ m to avoid under determination. Given a set of observed data samples in dataset X = [x_{1}, ⋯, x_{N}] with x_{t} = As_{t}, the ICA is to recover the independent source signals s_{1}, ⋯, s_{N}, based on a linear reconstruction model

y = Wx = WAs  (11.2)

where W = [w_{1}, ⋯, w_{n}] is a m×n reconstruction matrix, and y = [y_{1}, ⋯, y_{m}]^{T} contains the m reconstructed signals to match the independent source signals in s. Here ith component of y is y_{i} = w_{i}^{T} x where w_{i}^{T} is the ith row vector of W.

As ICA is an unsupervised method without training data for the ground truth regarding either A or s, we cannot find the reconstruction matrix W in closed form by the least squares method to minimize the squared error ||y−s||^{2}. Instead we have to find the optimal W based only on the desired requirement that the components in y = Wx need to be as independent of each other as possible.

The method of ICA can be compared with the PCA. Given the observed data x, both methods find a matrix W so that each data point as a vector x in the space is mapped to another vector y = Wx. In PCA, the transform matrix W = V^{T} (Eq. (9.18)) is the eigenvector matrix of the covariance matrix of x, i.e., the data points are projected onto the eigenvectors v_{i} as the orthogonal basis of the space, and components y_{i} = v_{i}^{T} x of y are decorrelated. (Only in the special case when y = [y_{1}, ⋯, y_{n}] is gaussian, are its components also independent.) While in ICA, W is in general not orthogonal, and the components are independent and necessarily also uncorrelated. We therefore see that the ICA is a more powerful method in the senese that it satisfies a stronger requirement for W.

A typical example of the problems that can be addressed by the ICA is the blind source separation (BSS) problem, i.e., to separate and reconstruct the linearly mixed source signals. The word “blind” means that no prior knowledge about sources or the mixing process is assumed available, except that the source signals are statistically independent. Although this BSS problem seems severely under constrained, the method of ICA can find nearly unique solutions satisfying certain properties. A typical example of the BBS problem is the “cocktail party problem”, where the goal is to reconstruct the individal voices of m speakers from n microphones (n ≥ m) recording all the conversations.

Here are some additonal facts about the source signal s and the observed data x = As:

• The number n for the observed variables in x must be no fewer than the number m for the independent source signals in s. We assume m = n in the following.

• The source components are stochastically independent and therefore uncorrelated, and they have to be non-Gaussian.

• The estimation of A and s is up to a scaling factor. Let

C = diag(c_{1}, ⋯, c_{n}), C^{−1} = diag(1/c_{1}, ⋯, 1/c_{n})  (11.3)

and A′ = AC^{−1} and s′ = Cs, we have

x = As = [AC^{−1}][Cs] = A′s′  (11.4)

• Due to the arbitrary scaling factor C, we can assume all the independent components have unit variance σ_{i}^{2} = E[s_{i}^{2}] = 1. As they are independent and necessarily also uncorrelated, we have σ_{ij}^{2} = E[s_{i}s_{j}] = δ_{ij}, i.e.,

E(ss^{T}) = I  (11.5)

• The estimated independent components are not in any particular order. When the order of the corresponding elements in both s and A is rearranged, x = As still holds.

The theoretical foundation of ICA algorithms is the central limit theorem, stating that the distribution of the sum (average or linear combination) of N independent random variables approaches Gaussian as N → ∞. For example, the face value of a dice has a uniform distribution from 1 to 6. But the distribution of the sum of a pair of dice is no longer uniform. It has a maximum probability at the mean of 7. As the number N of dice increases, the distribution of the sum of the face values will approach a Gaussian. More generally, let x_{1}, ⋯, x_{N} be N random variables independently drawn from an arbitrary distribution with mean μ and variance σ^{2}. Then the distribution of the mean x = ∑_{i=1}^{N} x_{i}/N approaches a Gaussian with mean μ and variance σ^{2}/N.

To solve the BSS problem, we want to find a matrix W so that y = Wx = WAs is as close to the independent components in the sources s as possible. This can be seen as the reverse process of the central limit theorem above. Consider the jth component y_{j} = w_{j}^{T} x_{j} of y, where w_{j}^{T} is the jth row of W. As a linear combination of all components of s, y_{i} is necessarily more Gaussian than any of the m source components {s_{1}, ⋯, s_{m}}. Therefore for y to be a good estimate of s, we desire to find W that maximizes the non-Gaussianity of y = Wx so that y is least Gaussian and most independent. The idea that non-Gaussianity is independence is in the core of the ICA method. Obviously if all source variables are Gaussian, the ICA method will not work.

All ICA algorithms can be considered as an optimization process that finds the matrix W that maximizes certain objective function (to be discussed below) that measures the non-Gaussianity of y = Wx, and thereby the independence of its components.

11.2 Preprocessing

Before considering specifically how to find the desired transform matrix W to reconstruct y = Wx to approximate the independent source signal s, we first consider some preprocessing process which is required for the FastICA algorthm discussed below.

The purpose of the preprocessing process is to convert the observed data x so that its components are of zero mean, unit variance, and independent of each other, i.e., its covariance matrix should be E[xx^{T}] = E[ss^{T}] = I, same as the source signal s.

• Centering

Subtract the mean E[x] from the observed variable x = As so it has zero mean. By doing so, the sources s also become zero mean because E[x] = A E[s] = 0 and E[s] can be estimated to be A^{−1}E[x].

• Whitening

Transform the observed variable x so that its components are uncorrelated and have unit variance. Do do so, we first find the eigenvalue matrix Λ and eigenvector matrix Φ of the covariance matrix E[xx^{T}] so that

Φ^{T} E[xx^{T}]Φ = Λ,  i.e., Λ^{−1/2}Φ^{T} E[xx^{T}]ΦΛ^{−1/2} = I (11.6)

As E[xx^{T}] is symmetric, Φ^{T} = Φ^{−1} is orthogonal. We now carry out a linear transform of the observed x so that it components are decorrelated (same as the KLT transform) and normalized with unity variance:

x′ = (Λ^{−1/2}Φ^{T})x,  i.e., x = (Λ^{−1/2}Φ^{T})^{−1}x′ = (ΦΛ^{1/2})x′ (11.7)

so that the covariance matrix of x′ becomes

E[x′x′^{T}] = E[(Λ^{−1/2}Φ^{T} x)(Λ^{−1/2}Φ^{T} x)^{T}] = E[Λ^{−1/2}Φ^{T} xx^{T} ΦΛ^{−1/2}]
= Λ^{−1/2}Φ^{T} E[xx^{T}]ΦΛ^{−1/2} = I  (11.8)

i.e., x′ is whitened with covariance σ_{ij}^{2} = δ_{ij}.

Once x is centered and whitened to become x′ = (Λ^{−1/2}Φ^{T})x, the reconstruction becomes

y = Wx = W(ΦΛ^{1/2})x′ = W′x′  (11.9)

where W′ = WΦΛ^{1/2} is the reconstruction matrix for whitened x′. We also have

E[x′x^{′T}] = E[(As)(As)^{T}] = AE[ss^{T}]A^{T} = AA^{T} = I  (11.10)

Also as y = W′x′ is to match s, we assume they have the same covariance E[yy^{T}] = E[ss^{T}] = I (Eq. (11.5)), and further get

I = E[yy^{T}] = E[(W′x′) (W′x′)^{T}] = W′E[x′x^{′T}]W^{′T} = W′W^{′T} (11.11)

after whitening, W′ becomes an orthogonal matrix.

We see that after whitening, both the mixing matrix A and the reconstruction matrix W′ becomes orthogonal and can therefore be interpreted as rotation matrices.

For convenience, in the following discussion we will still denote the whitened data samples by x and the reconstruction matrix by W.

11.3 Non-Gaussianity Maximization

In ICP, we desire to find the optimal W that maximizes the non-Gaussianity of the reconstructed y = Wx. In general, the non-Gaussianity of a random y with density function p(y) can be measured by the negentropy, defined as the difference between its entropy H(y) and the entropy of a Gaussian variable y_{G} with the same variance:

J(y) = H(y_{G}) − H(y) = C − H(y) ≥ 0  (11.12)

where C = H(y_{G}) is a constant independent of y. We note that J(y) is nonnegative as Gaussian distribution has the maximum entropy among all distributions with the same variance (Section 3.2.1). We see that maximizing non-Gaussianity measured by the negentropy J(y) is equivalent to minimizing the entropy H(y). This negentropy defined for a scalar variable y can be generalized to a random vector y = [y_{1}, ⋯, y_{m}]^{T}:

J(y) = H(y_{G}) − H(y) = C − H(y) ≥ 0  (11.13)

The non-Gaussianity of y = Wx can also be measured in terms of the mutual information (Eq. (3.186)) shared by its components {y_{1}, ⋯, y_{m}}. We desire to find the optimal W that maximizes the independence or minimizes the mutual information of the components of y = Wx:

I(y) = ∑_{i=1}^{n} H(y_{i}) − H(y) = ∑_{i=1}^{n} H(y_{i}) − log |det W| − H(x)  (11.14)

where H(y) = H(x)+log | det W| is the entropy of y = Wx as a linear function of x (Eq. (3.179)).

As after preprocessing W is orthogonal with det W = ±1 independent of W, and so is H(x), the mutual information can be written as

I(y) = ∑_{i=1}^{n} H(y_{i}) + C  (11.15)

where C is a constant representing the last two terms in the equation.

We see that minimizing all entropies H(y_{i}) = H(w_{i}^{T} x) is equivalent to minimizing both Gaussianity (i.e., maximizing the negentropy J(y), and the mutual information I(y). If the mutual information of a set of variables is decreased indicating the variables are less dependent, then the negentropy is increased, and components y_{i} are less Gaussian.

11.4 The FastICA Algorithm

The negentropy can be generalized to J(y) = H(y_{G})−H(y) for a random vector y = [y_{1}, ⋯, y_{m}]^{T}. In ICA, we desire to find matrix W so that y = Wx has the maximum negentropy and therefore is most non-Gaussian. The exact J(y) is difficult to find without the specific distribution p(y), but it can be approximated by

J(y) ≈ ∑_{i} k_{i}(E[G_{i}(y)] − E[G_{i}(g)])^{2}  (11.16)

where k_{i} are some positive constants, y is assumed to have zero mean and unit variance due to whitening, and g is a Gaussian variable also with zero mean and unit variance. G_{i} are some non-quadratic functions such as

G_{1}(u) = \frac{1}{a} log cosh (a u), G_{2}(u) = − exp(−u^{2}/2)  (11.17)

where 1 ≤ a ≤ 2 is some suitable constant. Although this approximation may not be accurate, it is always greater than zero except when x is Gaussian. In particular, when only a single term is used in Eq. (11.16), we have

J(y) = (E[G(y)] − E[G(g)])^{2}  (11.18)

Since the second term is a constant, we simply needs to maximize E[G(y)].

The algorithm FastICA is to find the transform matrix W so that y = Wx maximizes the following as an objective function representing the negentropy in Eq. (11.18) with the constant second term removed:

∑_{i=1}^{m} E[G(y_{i})] = ∑_{i=1}^{m} E[G(w_{i}^{T} x)]  (11.19)

We first consider computing w_{i} for y_{i} as one of the m components (with the subscript i dropped for now). This is a constrained optimization problem of finding the optimal vector w that maximizes the above treated as an objective function but subject to the constraint that it needs to be normalized. To solve this problem, we first construct the Lagrangian

L(w) = E[G(w^{T} x)] − \frac{μ}{2}(w^{T} w − 1)  (11.20)

and then set to zero the derivative of L(w) with respect to w:

f(w) = \frac{d}{dw} L(w) = E[x \frac{d}{dw} (w^{T} x)] − λw = E[x g(w^{T} x)] − μw = 0 (11.21)

where g(z) = dG(z)/dz is the derivative of function G(z). This equation system can be solved for w iteratively by the Newton-Raphson method:

w ⇐ w − J_{f}^{−1}(w)f(w)  (11.22)

where J(w) is the Jacobian matrix of f(w)

J_{f}(w) = \frac{d}{dw} f(w) = E[xx^{T} g′(w^{T} x)] − λI  (11.23)

where g′(z) is the derivative of g(z) or second derivative of G(z). We further approximate the first term on the right to get

E[xx^{T} g′(w^{T} x)] ≈ E[xx^{T}] E[g′(w^{T} x)] = E[g′(w^{T} x)]I  (11.24)

where E[xx^{T}] = I as the dataset is assumed to be whitened, then the Jacobian becomes diagonal

J_{F} (w) = (E[g′(w^{T} x)] − λ) I  (11.25)

and the Newton-Raphson iteration becomes:

w_{n+1} = w_{n} − \frac{1}{E[g′(w_{n}^{T} x)] − μ} (E[x g(w_{n}^{T} x)] − μw)  (11.26)

Multiplying both sides by the scaler λ − E[g′(w^{T} x)], we get

w_{n+1} = E[xg(w_{n}^{T} x)] − E[g′(w_{n}^{T} x)]w  (11.27)

where the scaled version of w_{n+1} is still denoted by w as it is to be normalized right away anyway.

This algorithm can then be repeated to get all row vectors in W each for one of the m row vectors w_{i}. However, if this process is carried out independently, the resulting vectors may not be orthogonal to each other. To address this issue, the Gram-Schmidt method in Section A.2 can be used. Specifically, during each iteration of the computation for w_{i} for i > 1, before it is normalized, we also subtract its projection onto each of the previously found w_{1}, ⋯, w_{i−1} (Eq. ??GramSchmidtIteration)), so that it is guaranteed to be orthogonal to all of them. All vectors such computed will guarantee that the final matrix W is orthogonal satisfying W^{T} = W^{−1} as required.

Now we have the following FastICA algorithm that is carried out m times each for one of the vectors w_{i}, (i = 1, ⋯, m):

1. Choose an initial random guess for w_{i}

2. Iterate:

w = E[xg(w^{T} x)] − E[g′(w^{T} x)]w  (11.28)

where the expectation is approximated by the average of all N samples. 3.

w ⟵ w − ∑_{j=1}^{i−1}(w^{T} w_{j})w_{j}  (11.29)

4. Normalize:

w ⟵ w/||w||  (11.30)

5. If not converged, go back to step 2.

The Matlab code segment below shows the essential part of the FastICA algorithm for ICA problem, where Imax is the maximum number of iterations, and tol is the tolerance, e.g., 10^{−6}.

syms u
f=-exp(-u^2/2);
f=log(cosh(u));
g=matlabFunction(diff(f));
dg=matlabFunction(diff(diff(f)));
[d N]=size(X);         % size of dataset
[X T]=whiten(X);       % whiten dataset
W=[];                  % initialize matrix W
one=ones(N,1);
for i=1:d              % for each vector w
    w=rand(d,1);       % random initialization
    for i=1:Imax
        wold=w;
        wx=w'*X;
        w=X*g(wx)'/N-dg(wx)*one*w/N;
        if i>1
            w=w-W*W'*w;
        end
        w=w/norm(w);
        if norm(w-wold)<tol
            break
        end
    end
    W=[W w];
end
Y=W'*X;

W=W*T;

Here is the whitening function that takes the dataset X as input and generates the whitened data and the whitening transform matrix as the output:

function [X T]=whitening(X)
   [d N]=size(X);
   X=X-repmat(mean(X,2),1,N);
   [U S V]=svd(X);
   T=sqrt(N)*diag(1./(diag(S(1:d,1:d))))*U';
   X=T*X;
end

11.5 Likelihood Maximization

The ICA problem can also be addressed by the method of maximum likelihood estimation. We first consdier the following two ways to estimate the unknown mixing matrix A.

• Matrix A can be estimated so that p(x|A) of x = As conditioned on A matches the actual p(x) based on the observed data, i.e., the KL-divergence D_{KL}(p(x)||p(x|A) is minimized. Here p(x|A) is related to p(s) = p(s_{1}, ⋯, s_{m}) = ∏_{i=1}^{m} p_{i}(s_{i}) of the independent source signals by (Eq. (3.76)):

p(x|A) = | det A^{−1}| p(s) = | det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{i})  (11.31)

Now the KL-divergence to be minimized can be obtained as

D_{KL}(p(x)||p(x|A) = ∫ p(x) log \frac{p(x)}{p(x|A)}dx
= ∫ p(x log p(x)dx − ∫ p(x) log (| det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{i})) dx

= H(x) − log | det A^{−1}| ∫ p(x)dx − ∫ p(x) ∑_{i=1}^{m} log p_{i}(s_{i})dx

= H(x) − log | det A^{−1}| − E_{x} [∑_{i=1}^{m} log p_{i}(s_{i})]  (11.32)

where s_{i} is the ith component of s = A^{−1}x and the expectation is with respect to observed data x and can be estimated as the average of the samples in the dataset X = [x_{1}, ⋯, x_{N}]. Here the first term H(x) independent of A can be dropped.

• Matrix A can also be estimated by maximizing its likelihood given the observed dataset X containing N i.i.d. samples:

L(A|X) = p(X|A) = ∏_{n=1}^{N} p(x_{n}|A) = ∏_{n=1}^{N} | det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{in})

= | det A^{−1}|^{N} ∏_{n=1}^{N} ∏_{i=1}^{m} p_{i}(s_{in})  (11.33)

where p(x_{n}|A) is given in Eq. (11.31). The log-likelihood is:

l(A|X) = log L(A|X) = log [| det A^{−1}|^{N} ∏_{n=1}^{N} ∏_{i=1}^{m} p_{i}(s_{in})]

= N [log | det A^{−1}| + \frac{1}{N} ∑_{n=1}^{N} ∑_{i=1}^{m} p_{i}(s_{in})]

∝ log | det A^{−1}| + E_{x} [∑_{i=1}^{m} log p_{i}(s_{i})]  (11.34)

Here in the last step the average over all N data samples in the second term is treated as the estimated expectation.

Comparing these two cases, we note that maximizing the log-likelihood in Eq. (11.34) is equivalent to minimizing the KL-divergence in Eq. (11.32), as the two expressions are negation of each other.

To find the optimal W so that the reconstructed signel y ideally matches the source signal s:

y = Wx = WAs = s  (11.35)

We can simply let W = A^{−1}, i.e., y_{i} = w_{i}^{T} x = s_{i} for all i = 1, ⋯, m. Then either Eq. (11.32) or (11.34) can be used as the objective function for the optimal W, when A^{−1} is replaced by W and s_{i} is replaced by w_{i}^{T} x_{i}:

J(W) = log | det W| + E [∑_{i=1}^{m} log p_{i}(w_{i}^{T} x)]  (11.36)

Its gradient with respect to W is:

g_{J}(W) = \frac{d}{dW} J(W) = \frac{d}{dW} log | det W| + E [\frac{d}{dW} ∑_{i=1}^{m} log p_{i}(w_{i}^{T} x)] (11.37)

The first term is

\frac{d}{dW} log | det W| = W^{−T}  (11.38)

where W^{−T} = (W^{T})^{−1} = (W^{−1})^{T}. For each term inside the summation of the second term, we find its derivative with respect to the component w_{ki} in the kth row and ith column of W:

\frac{d}{d w_{ki}} log p_{i}(w_{i}^{T} x) = (\frac{p′_{i}}{p_{i}} ) x_{k} = ϕ_{i}x_{k}  (11.39)

where function ϕ(u) is defined as

ϕ(u) = \frac{d}{du} log p(u) = \frac{1}{p(u)} \frac{d}{du}p(u)  (11.40)

based on which the matrix form of the second term can be written as ϕx^{T}, where ϕ = [ϕ_{1}, ⋯, ϕ_{m}]^{T}. Now we have

g_{J}(W) = W^{−T} + E[ϕx^{T}]  (11.41)

and the optimal W that maximizes J(W) for the log-likelihood can be found iteratively based on the gradient ascend method:

W_{n+1} = W_{n} + δg_{J}(W_{n}) = W_{n} + δ [W^{−T} + E[ϕx^{T}]]  (11.42)

By post-multiplying W^{T} W the gradient g_{J}(W) we get the natural gradient along a modified search direction in each step of the iteration:

W_{n+1} = W_{n} + δg_{J}(W_{n}) = W_{n} + δ [I + E[ϕWx^{T}]] W  (11.43)

which becomes more efficient and converges more quickly.

If the stochastic gradien method is used, the iterative update of W is baased on only one of the N samples at a time, instead of the average of all N of them, i.e., the expectation symbol E can be dropped.

Various forms of p(u) and ϕ(s) = p′(u)/p(u) have been suggested including the following:

p(u) = \frac{1}{cosh(u)}, ϕ(u) = \frac{1}{p(u)} \frac{d}{du}p(u) = − tanh(u)  (11.44)

The Matlab code segment below shows the essential part of the MLE method for ICA problem, where Imax is the maximum number of iterations, tol is the tolerance, e.g., 10^{−6}. We note that whitening is not required for the algorithm and is therefore optional.

    [d N]=size(X);      % size of the dataset
    T=eye(d);           % whitening transform matrix
    if whiten
        [X T]=whitening(X);
    end
    W=eye(d);           % initialize W
    for i=1:Imax
        Y=W*X;
        Wnew=W+eta*(W-tanh(Y)*(Y'*W)/N);

if mean(abs(Wnew(:)-W(:))) < tol
    break
end
W=Wnew;
end
W=W*T;

Part IV Classification 12 Statistic Classification

12.1 Discriminative vs. Generative Methods for Classification

As one of the most important tasks in machine learning, pattern classification is to classify some objects of interest, generically referred to as patterns and described by a set of d features or attributes that characterizes the patterns, to one of some K classes or categories. Each pattern is represented by a vector (or a point) x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional feature space, where x_{i} (i = 1, ⋯, d) is a variable for the measurement of the ith feature. Symbolically, the K classes can be denoted {C_{1}, ⋯, C_{K}}, and a pattern x belonging to the kth class is denoted by x ∈ C_{k}. Pattern classification can therefore be considered as the process by which the d-dimensional feature space is partitioned into K regions each corresponding to one of the K classes. The boundaries between these regions, called decision boundaries, are to be determined by the specific algorithm, called a classifier, used for the classification.

Pattern classification can be carried out as either a supervised or unsupervised learning process, depending on the availability of a training set containing patterns of known class identities. Specifically, the training set contains a set of N patterns in X = [x_{1}, ⋯, x_{N}], labeled respectively by the corresponding component in y = [y_{1}, ⋯, y_{N}]^{T} representing the class identities of the corresponding patterns in some way. For example, we can use y_{k} ∈ {1, ⋯, K} to indicate x_{k} ∈ C_{y_{k}}. In the special case when K = 2, there are only two classes C_{1} = C_{+} and C_{0} = C_{−}, and the classifier becomes binary based on training pattern x_{j} (j = 1, ⋯, N), each labeled by y_{j} = 1 if x_{j} ∈ C+ or y_{j} = −1 if x_{j} ∈ C_{−}.

We assume there are N_{k} training samples {x_{1}^{(k)}, ⋯, x_{N_{k}}^{(k)}} all labeled to belong to C_{k}, (k = 1, ⋯, K), and in total N = ∑_{k=1}^{K} N_{k} samples in the training set. If the training set is a fair representation of all patterns of different classes in the entire dataset, then P_{k} = N_{k}/N can be treated as an estimate of the a priori probability that any randomly selected pattern x happens to belong to class C_{k}, without any prior knowledge of the pattern.

Once a classifier is properly trained according to a specific algorithm based on the traning set, the feature space is partitioned into regions corresponding to the different classes and any unlabeled pattern of unknown class as a vector x in the feature space can be classified into one of the K classes.

Supervised classification can be considered as a process of establishing the corresponding relationship between the patterns x_{1}, ⋯, x_{N} treated as the independent or input variables to the classifier, and the classes C_{1}, ⋯, C_{K} the input patterns belong, treated as the dependent or output variables. Therefore regression and classification can be considered as the same supervised learning process: modeling the relationship between the data points in {x_{1}, ⋯, x_{N}} and their corresponding labelings (or targets) in {y_{1}, ⋯, y_{N}}. This process is regression when the labelings take continous real values, but it is classification when they are discrete categorical representing different classes. Some methods in the previous chapter on regression analysis are actually used as classifiers, such as logistic and solfmax regressions, and the method of Gaussian process can also be used for classification.

If the training data of labeled patterns are unavailable, various unsupervised learning methods can be used to assign each unlabeled patterns into one of the K different groups, called clusters, according to its position in the feature space, based on the overall spatial structure and distribution of the data set in the feature space. This process is called clustering analysis or simply clustering.

There exsit a variety of methods for learning, including both regression and classification, based on different models assumed. One way to characterize these methods is to put them all in a probabilistic framework, in terms of the probabilities of the given dataset incuding data points X and the corresponding labeling y. Now a method can be categorized into either of the following two groups:

• Discriminative model (conditional or backward model)

A discriminative method stablishes a model that maps a data pont x to a class labeling y. Such a model is either a pure or traditional discriminative model if it is deterministic and aims to fit the training set in some optimal way, or a conditional model if it is probabilistic in nature, such as the conditional probability p(y|x). The model parameters in θ are obtained in some optimal way based on the training set. Then prediction can be made for any unlabeled x in terms of the corresponding y. As a discriminative method aims at finding the decision boundaries between different classes based on the training set, only those data samples that are close to the boundaries play an important role, while all other samples farther away from the boundaries are mostly ignored. Typical discrimiative methods include:

– K-nearest neighbors algorithm

– Linear and logistic regressions

– Support vector machines

– Perceptrons and Neural networks

– Decision trees and random forests

• Generative model (forward model)

A generative method first assumes certain probabilistic model for the underlying structure of the observed data, such the joint probability p(x, y|θ) based on all data samples available. It then estimates the parameter θ of the model based on the training dataset, and obtains the conditional probability p(y|x) (by Bayes’ theorem) based on which a prediction can be made for any unlabeled x to find the corresponding y. As in general the generative method is based on some probabilistic model of the data, it can be used for unsupervised learning as well as supervised.

Typical generative methods include:

– Naive Bayes classifiers

– Gaussian mixture classifiers

– Hidden Markov model

Here are some comparisons between the two approaches:

• The discriminative methods find the decision boundary in the feature space directly based on the data points in the training set, in general they

– are simpler than the generative approach requiring problem-specific knowledge for building models of the data,

– are effective in producing accurate result when the dataset is large

– provide no insight or interpretation regarding the data and no uncer-

tainty estimate

• The generative methods first establish a probabilistic model for the underlying structure of the data as an effort to explain how the data was generated, and then finds the decision boundary based the model. In general, they – allow the use of problem-specific knowlege for building the model,

– can provide explanation and interpretation of the data,

– may be less prone to overfitting than a discriminative method,

– can provide uncertainty estimate

– may not be as accurate as the discriminative methods if the model does not fit the dataset well.

12.2 K Nearest Neighbor and Minimum Distance Classifiers

Here we first consider a set of simple supervised classification algorithms that assign an unlabeled sample x to one of the K known classes based on set N of training samples X = [x_{1}, ⋯, x_{N}], where each sample x_{n} is labeled by y_{n} = k ∈ {1, ⋯, K}, indicating it belongs to class C_{k}.

• k Nearest neighbors (k-NN) Classifier

Given an unlabeled pattern x, we first find its k nearest neighbors in the training dataset, and then assign x to one of the K classes by a majority vote of the k neighbors based on their class labelings. The voting can be weighted so that closer neighbors are more heavily weighted than those that are farther away. In particular, when k = 1, x is assigned to the class of its closest neighbor.

While the k-NN method is simple and straight forward, its computational cost is high as classifying any unlabeled pattern x requires computing distances to all data points in the training set.

• Minimum Distance Classifier

Given a set of training data points {x_{1}, ⋯, x_{N_{k}}} all belonging to the kth class C_{k} (k = 1, ⋯, K), we can find their mean and covariance to represent the class:

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N_{k}} x_{n},  Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N_{k}}(x_{n} − m)^{T} (x_{n} − m)  (12.1)

Any unlabeled data point x can be classified to one of the K classes based on certain distance d(x, C_{k}) between x and each of class C_{k}:

if d(x, C_{k}) = min{d(x, C_{i}) i = 1, ⋯, C} then x ∈ C_{k}  (12.2)

We could simply use the Euclidean distance d_{E}(x, m_{k}) between x and m_{k}. But such a classification may not reliable as Euclidean distance does not take into consideration the covariance Σ_{k} representing how the N_{k} samples are distributed in the feature space, as illustrated by the following example.

Example 1: As illustrated in the figure below (left plot), a point x = 1 in 1-D space is to be classified into one of the two classes represented by their corresponding Gaussian pdfs:

C_{1} ∼ N (5, 1.2^{2}),  C_{2} ∼ N (−5, 3^{2})  (12.3)

If only the two means m_{1} = 5 and m_{2} = −5 are considered, we have d_{E}(x, m_{1}) = 4 < d_{E}(x, m_{2}) = 6, i.e., x = 1 is closer to m_{1} than m_{2} and therefore should be classified to class C_{1}. However, as shown in the plot, x should be classified to class C_{2}, if the variances σ_{1} = 1.2 and σ_{2} = 3 are also taken into consideration.

We see the distance d(x, C_{k}) should be positively related to d_{E}(x, m_{k}), but inversely related to σ_{k}, i.e., we can define d(x, C_{k}) = (x − m_{k})^{2}/σ_{k}^{2}. Based on this distance, we find d(x_{1}, m_{1}) = 11.11 > d(x_{1}, m_{2}) = 4.0, i.e., x = 1 should be classified to C_{2}.

In a higher dimensional feature space, we can carry out classification based on the more generally defined Mahalanobis distance between a point x and a distribution represented by m_{k} and Σ_{k}:

d_{M}(x, C_{k}) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})  (12.4)

Example 2: As illustrated in the above figure (right plot), two samples x_{1} = 1 and x_{2} = 3 are to be classified into either of two classes:

C_{1} ∼ N (0, 1.2^{2}),  C_{2} ∼ N (0, 3^{2})  (12.5)

As the two means m_{1} = m_{2} = 0 are the same, |x_{i} − m_{1}|^{2} = |x_{i} − m_{2}|^{2} for both samples x_{1} and x_{2}, they are both classified into C_{2} with a greater variance σ_{2} > σ_{2} therefore smaller Mahalanobis distances:

\{ \begin{matrix}d_{M}(x_{1}, C_{1}) = \frac{(x_{1}−m_{1})^{2}}{σ_{1}^{2}} = \frac{1}{1.2^{2}} = 0.69,  d_{M}(x_{1}, C_{2}) = \frac{(x_{1}−m_{1})^{2}}{σ_{2}^{2}} = \frac{1}{9} = 0.11 \\ d_{M}(x_{2}, C_{1}) = \frac{(x_{1}−m_{1})^{2}}{σ_{1}^{2}} = \frac{3^{2}}{1.2^{2}} = 6.25,  d_{M}(x_{1}, C_{2}) = \frac{(x_{1}−m_{1})^{2}}{σ_{2}^{2}} = \frac{3^{2}}{9} = 1.00 \end{matrix}

(12.6)

However, as shown in the plot, x_{1} = 1 should be classified to C_{1}. We therefore see that sometimes the Mahalanobis distance is not reliable for classification, and some better method need to be considered, as discussed later.

12.3 Naive Bayes Classification

The method of naive Bayes (NB) classification is a classical supervised classification algorithm, which is first trained by a training set of N samples X = [x_{1}, ⋯, x_{N}] and their corresponding labelings y = [y_{1}, ⋯, y_{N}]^{T}, and then used to classify any unlabeled sample x into class C_{k} with the maximumm posterior probability. As indicated by the name, naive Bayes classification is based on Bayes’ theorem:

P (C_{k}|x) = \frac{p(x, C_{k})}{p(x)} = \frac{p(x|C_{k})P (C_{k})}{p(x)}  (12.7)

where

• P (C_{k}) is the prior probability that any data sample belongs to class C_{k} without observing its values, more briefly denoted by P_{k}, which can be estimated by

P_{k} = \frac{N_{k}}{N} = \frac{N_{k}}{∑_{l=1}^{K} N_{l}} , (k = 1, ⋯, K) satisfying ∑_{k=1}^{K} P_{k} = 1

(12.8)

where N_{k} is the number of data samples in the training set labeled to belong to class C_{k}. This estimation is based on the assumption that the training samples are evenly drawn from the entire population, and are therefore a fair representation of all K classes.

• p(x|C_{k}) = L(C_{k}|x) is the likelihood for any observed x to belong to class C_{k}, which is the conditional probability of x given that x ∈ C_{k}, assumed to be a Gaussian in terms of the mean vector m_{k} and covariance matrix Σ_{k}:

p(x|C_{k}) = N(x, m_{k}, Σ_{k}) = \frac{1}{(2π)^{d/2} |Σ_{k}|^{1/2}} exp [−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})]

(12.9)

This assumption is based on the fact that the Gaussian distribution has the maximum entropy (uncertainty) among all probability density functions with the same covariance, i.e., it imposes the least amount of unsupported constraint on the model for the dataset.

• p(x, C_{k}) is the joint probability of x and C_{k}:

p(x, C_{k}) = p(x|C_{k})P (C_{k}) = P (C_{k}|x)p(x)  (12.10)

• p(x) is the distribution of any data sample in the dataset, independent of its classe identity, the weighted sum of all likelihood p(x|C_{k}) for k = 1, ⋯, K, i.e., the joint probability p(x, C_{k}) marginalized over all K classes:

p(x) = ∑_{k=1}^{K} p(x, C_{k}) = ∑_{k=1}^{K} P_{k} p(x|C_{k}) = ∑_{k=1}^{K} P_{k} [\frac{1}{(2π)^{d/2} |Σ_{k}|^{1/2}} exp (−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}))]

(12.11)

• P (C_{k}|x) is the posterior probability that a data sample x belongs to class C_{k} based on the observed values in x = [x_{1}, ⋯, x_{d}]^{T}.

Based on Bayes’ theorem discussed above, the naive Bayes method classifies an unlabeled data sample x to class C_{k} with the maximum posterior probability (maximum a posteriori (MAP)):

if P (C_{k}|x) = max_{l}{P (C_{l}|x), (l = 1, ⋯, K)}, then x ∈ C_{k}  (12.12)

As p(x) is common to all K classes (independent of k), it plays no role in the relative comparison among the K classes, and can therefore be dropped, i.e., x is classified to C_{k} with maximal p(x|C_{k}) P_{k}.

The naive Bayes classifier is an optimal classifier in the sense that the classification error is minimum. To illustrate this, consider an arbitrary boundary between two classes C_{1} and C_{2} that partitions the 1-D feature space into two regions R_{1} and R_{2}, as shown below. The probability of a misclassification is the joint probability P (x ∈ C_{i} ∩ x ∈ R_{j}) for a sample x ∈ C_{i} but falling in R_{j}. The total probability of error due to misclassification can be expressed as:

P (error) = P ((x ∈ R_{2}) ∩ (x ∈ C_{1})) + P ((x ∈ R_{1}) ∩ (x ∈ C_{2}))
= P (x ∈ R_{2}/C_{1}) P_{1} + P (x ∈ R_{1}/C_{2}) P_{2}
= P_{1} ∫_{R_{2}} p(x/C_{1})dx + P_{2} ∫_{R_{1}} p(x/C_{2})dx  (12.13)

It is obvious that the Bayes classifier is indeed optimal, due to the fact that its boundary p(x|C_{i})P_{i} = p(x|C_{j})P_{j} between classes C_{i} and C_{j} (the bottom plot) guarantees the classification error (shaded area) to be minimum.

To find the likelihood function p(x|C_{k}) = N (x, m_{k}, Σ_{k}), we need to find m_{k} and Σ_{k} based on the training set D = {X, y}, based on the N_{k} training samples in class C_{k} by the method of maximum likelihood estimation

m_{k} = \frac{1}{N_{k}} ∑_{i=1}^{N_{k}} x_{i}, Σ_{k} = \frac{1}{N_{k}} ∑_{i=1}^{N_{k}}(x_{i} − m_{k})(x_{i} − m_{k})^{T} , (x_{i} ∈ C_{k}) (12.14)

Once both P_{k} = N_{k}/N and p(x|C_{k}) = N (x, m_{k}, Σ_{k}) are available, the classifier is trained, and the posterior P (C_{k}|x) can be calculated for the classification in Eq. (12.12).

As the classification is based on the relative comparison of the posterior P (C_{k}|x), any monotonic mapping of the posterior can be equivalently used to simplify the computation, such as the logarithmic function:

log P (C_{k}|x) = log [p(x|C_{k})P_{k}/p(x)] = log p(x|C_{k}) + log P_{k} − log p(x)
= −\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) − \frac{N}{2} log(2π) − \frac{1}{2} log |Σ_{k}| + log P_{k} − log p(x) (12.15)

Dropping the constant terms log p(x) and N log(2π)/2 that are independent of the index k and therefore play no role in the comparison above, we get the quadratic discriminant function:

D_{k}(x) = −\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) − \frac{1}{2} log |Σ_{k}| + log P_{k}  (12.16)

Now the classification can be represented by

if D_{l}(x) = max{D_{k}(x), (k = 1, ⋯, K)}, then x ∈ C_{l}  (12.17)

Equivalently, we can also treat the negative of this discriminant function as a distance measurement:

d_{k}(x) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) + log |Σ_{k}| − 2 log P_{k}  (12.18)

for minimum distance classification. We note that the first term is the Mohalanobis distance defined previously. However, the additonal terms in the expression contribute to better classification performance.

Geometrically, the feature space is partitioned into regions corresponding to the K classes by the decision boundaries between every pair of classes C_{i} and C_{j}, represented by the equation D_{i}(x) = D_{j}(x), i.e.,

−\frac{1}{2}(x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i}) − \frac{1}{2} log |Σ_{i}| + log P_{i}

= −\frac{1}{2}(x − m_{j})^{T} Σ_{j}^{−1}(x − m_{j}) − \frac{1}{2} log |Σ_{j}| + log P_{j}  (12.19)

which can be written as

x^{T} Wx + w^{T} x + w = 0  (12.20)

where

W = −\frac{1}{2}(Σ_{i}^{−1} − Σ_{j}^{−1})
w = Σ_{i}^{−1}m_{i} − Σ_{j}^{−1}m_{j}
w = −\frac{1}{2}(m_{i}^{T} Σ_{i}^{−1}m_{i} − m_{j}^{T} Σ_{j}^{−1}m_{j}) − \frac{1}{2} log \frac{|Σ_{i}|}{|Σ_{j}|} + log \frac{P_{i}}{P_{j}} (12.21)

In general, this decision boundary is a quadratic hypersurface (hypersphere, hyperellipsoid, hyperparabola, or hyperhyperbola), by which an unlabeled data sample x is classified into either of the two classes C_{i} and C_{j} based on whether it is on the positive or negative side of the surface::

if x^{T} Wx + w^{T} x + w \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then x ∈ \{ \begin{matrix}C_{i} \\ C_{j} \end{matrix}  (12.22)

We further consider several special cases:

• All classes have the same prior:

P_{i} = P_{j} (i, j = 1, ⋯, K)  (12.23)

then the last term of D_{i}(x) is zero, and we have

D_{i}(x) = −\frac{1}{2}(x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i}) − \frac{1}{2} log |Σ_{i}|  (12.24)

We can reconsider Example 2 in the previous section but now based on the negative of the discriminant function above treated as a distance d(x, C_{i}) = (x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i}) + log |Σ_{i}|, and get

\{ \begin{matrix}d(x_{1}, C_{1}) = 0.69 + log(1.2^{2}) = 1.06,  d_{M}(x_{1}, C_{2}) = 0.11 + log(3^{2}) = 2.31 \\ d_{M}(x_{2}, C_{1}) = 6.25 + log(1.2^{2}) = 6.61,  d_{M}(x_{1}, C_{2}) = 1 + log(3^{2}) = 3.20 \end{matrix}

(12.25)

Now we see that x_{1} is classified into class C_{1}, while x_{2} into C_{2}, as desired. The misclassification of x_{1} to C_{2} by the Mahalanobis distance, the first term, is corrected, due to the addition of the second term log(σ^{2}). The plot below shows the partitioning of the 1-D space for the two classes C_{1} and C_{2}:

• All classes have the same covariance matrix Σ_{i} = Σ, the discriminant function becomes:

D_{i}(x) = −\frac{1}{2}(x − m_{i})^{T} Σ^{−1}(x − m_{i}) + log P_{i}  (12.26)

Moreover, if all P_{i} are the same and the second term is dropped, D_{i}(x) becomes the negative Mahalanobis distance.

The boundary equation D_{i}(x) = D_{j}(x) between C_{i} and C_{j} becomes

−\frac{1}{2}(x − m_{i})^{T} Σ^{−1}(x − m_{i}) + log P_{i} = −\frac{1}{2}(x − m_{j})^{T} Σ^{−1}(x − m_{j}) + log P_{j}

(12.27)

As the quadratic terms x^{T} Σ^{−1}x on both sides of the equation are the same, it becomes a linear equation:

w^{T} x + w = 0  (12.28)

where

w = Σ^{−1}(m_{i} − m_{j})  (12.29)

and

w = −\frac{1}{2}(m_{i}^{T} Σ^{−1}m_{i} − m_{j}^{T} Σ^{−1}m_{j}) + log \frac{P_{i}}{P_{j}}  (12.30)

This linear equation represents a hyperplane between the two points m_{i} and m_{j} and perpendicular to the straight line Σ^{−1}(m_{i} − m_{j}) (the straight line m_{i} − m_{j} rotated by matrix Σ^{−1}).

• All classes have the same isotropic distribution:

Σ_{i} = σ^{2} I = diag[σ^{2}, ⋯, σ^{2}]  (12.31)

then |Σ_{i}| = σ^{2d} and D_{i}(x) becomes

D_{i}(x) = −\frac{||x − m_{i}||^{2}}{2σ^{2}} + log P_{i}  (12.32)

Note that the term log |Σ_{i}| has been dropped from the original expression of D_{i}(x) as it is now the same for all classes.

The boundary equation D_{i}(x) = D_{j}(x) becomes:

\frac{||x − m_{i}||^{2}}{2σ^{2}} − log P_{i} = \frac{||x − m_{j}||^{2}}{2σ^{2}} − log P_{j}  (12.33)

which can be simplified to a linear equation:

w^{T} x + w = 0  (12.34)

where

w = m_{i} − m_{j}, w = −(m_{i}^{T} m_{i} − m_{j}^{T} m_{j}) + 2σ^{2} log \frac{P_{i}}{P_{j}}  (12.35)

This linear equation represents a hyperplane between the two points m_{i} and m_{j} and perpendicular to the straight line passing through these points.

• Further, if all classes have the same P_{i}, D_{i}(x) becomes

D_{i}(x) = −(x − m_{i})^{T} (x − m_{i}) = −||x − m_{i}||^{2}  (12.36)

and the Bayes classifier becomes minimum distance classifier based on Euclidean distance (maximizing D_{i}(x) is equivalent to minimizing ||x−m_{i}||^{2}.

The Matlab functions for training and testing are listed low:

function [M S P]=NBtraining(X,y)        % naive Bayes training
[d N]=size(X);               % dimensions of dataset
K=length(unique(y));                % number of classes
M=zeros(d,K);                       % mean vectors
S=zeros(d,d,K);                     % covariance matrices
P=zeros(1,K);                       % prior probabilities
for k=1:K
   idx=find(y==k);                 % indices of samples in kth class
   Xk=X(:,idx);                    % all samples in kth class
   P(k)=length(idx)/N;             % prior probability
   M(:,k)=mean(Xk');               % mean vector of kth class
   S(:,:,k)=cov(Xk');              % covariance matrix of kth class
end

end
function yhat=NBtesting(X,M,S,P)        % naive Bayes testing
    [d N]=size(X);                      % dimensions of dataset
    K=length(P);                        % number of classes
    for k=1:K
        InvS(:,:,k)=inv(S(:,:,k));      % inverse of covariance matrix
        Det(k)=det(S(:,:,k));           % determinant of covariance matrix
    end
    for n=1:N                           % for each of N samples
        x=X(:,n);
        dmax=-inf;
        for k=1:K
            xm=x-M(:,k);
            d=-(xm'*InvS(:,:,k)*xm)/2;
            d=d-log(Det(k))/2+log(P(k));  % discriminant function
            if d>dmax
                yhat(n)=k;              % assign nth sample to kth class
                dmax=d;
            end
        end
    end
end

The classification result in terms of the estimated class identity ŷ of the training samples compared with the ground truth class labeling y can be represented in the form of a K × K confusion matrix, of which the element in the ith row and jth column is the number of training samples labeled as a member of the ith class but actually classified into the jth class. For an ideal classifier that classifies all data samples correctly, this confussion matrix should be a diagonal matrix. But for a nonideal classifier, its error rate can be found as the percentage of the number of misclassified samples (sum of all off-diagonal elements) out of the total number of samples. The code below shows the function that generates such a confussion matrix based on the estimated class identity ŷ and the ground truth labeling y:

function [Cm er]=ConfusionMatrix(yhat,ytrain)
    N=length(ytrain);                 % number of test samples
    K=length(unique(ytrain));         % number of classes
    Cm=zeros(K);                      % the confussion matrixxs
    for n=1:N
        i=ytrain(n);
        j=yhat(n);
        Cm(i,j)=Cm(i,j)+1;
    end

    er=1-sum(diag(Cm))/N;             % error percentage
end

Example 1:

This example shows the classification of two cocentric classes with one surrounding the other. They are separated by an ellipse with 9 out of 200 samples misclassified, i.e., the error rate of 9/200 = 4.5%. The confusion matrix is shown below:

[ \begin{matrix}92 & 8 \\ 1 & 99 \end{matrix} ]  (12.37)

indicating 8 of the 100 samples belonging to class 1 are misclassified to class 2, and 1 of the 100 samples belonging to class 2 is misclassified to class 1.

Example 2:

This example shows the classification of a 2-class exclusive-or (XOR) data set with significant overlap, in terms of the confusion matrix and the error rate of 61/400 = 15.25%. Note that the decision boundaries are a pair of hyperbolas. The confusion matrix is

[ \begin{matrix}175 & 25 \\ 36 & 164 \end{matrix} ]  (12.38)

Example 3:

The first two panels in the figure below show three classes in the 2-D space, while the third one shows the partitioning of the space corresponding to the three classes. Note that the boundaries between the classes are all quadratic. The confusion matrix of the classification result is shown below, with the error rate 49/600 = 8.17%.

[ \begin{matrix}196 & 3 & 1 \\ 0 & 191 & 9 \\ 3 & 33 & 164 \end{matrix} ]  (12.39)

Example 4

The figure below shows some sub-samples of ten sets digits from 0 to 9, each is hand-written 224 times by different students. Each hand-written digit is represented as an 16 by 16 image containing 256 pixels, treated as a 256 dimensional vector.

The dataset can be visualized by applying the KLT transform to map all data points in the 256-D space into a 3-D space spanned by the three eigenvectors corresponding to the three greatest eigenvalues of the covariance matrix of the dataset, as shown below:

The naive Bayes classifier is applied to the dataset of handwritten digits from 0 to 9, of which each data sample is a 256-dimensional vector composed of 16 × 16 pixels of an image for the handwritten of a digit. The dataset contains 224 samples per class and N = 2240 in total for all K = 10 classes.

We note that the 256-dimensional covariance matrix Σ_{k} for each of the K = 10 classes is estimated by the 224 samples in the class, i.e., the estimated covariance matrix does not have a full rank and is not invertible. This issue can be resolved by applying KLT to the dataset to reduce its dimensionality from 256 to some value smaller than 224, such as 100.

The classifier is trained on half of the dataset of 1120 randomly selected samples and then tested on the remaining 1120 samples. The classification results are given below in terms of the confusion matrices of the traing samples (left) and the testing samples (right). All 1120 training samples are correctly classfied, while out of the remaining 1120 test samples, 198 are misclassified (17.7% error).

[ \begin{matrix}107 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 105 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 118 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 110 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 120 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 128 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 104 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 109 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 111 \end{matrix} ] , [ \begin{matrix}101 & 0 & 0 & 0 & 0 & 0 & 0 & 2 & 0 & 0 \\ 0 & 58 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 105 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 5 & 2 & 6 & 101 & 2 & 4 & 0 & 28 & 4 & 4 \\ 0 & 15 & 0 & 0 & 102 & 0 & 0 & 4 & 0 & 9 \\ 5 & 4 & 1 & 1 & 0 & 99 & 1 & 1 & 2 & 1 \\ 0 & 17 & 1 & 0 & 2 & 0 & 95 & 2 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 62 & 0 & 0 \\ 6 & 15 & 6 & 4 & 6 & 1 & 0 & 18 & 109 & 9 \\ 0 & 3 & 0 & 0 & 2 & 0 & 0 & 2 & 0 & 90 \end{matrix} ]

(12.40)

12.4 AdaBoost

The Adaptive boosting (AdaBoost) is a supervised binary classification algorithm based on a training set {(x_{i}, y_{i}) | i = 1, ⋯, N}, where each sample x_{i} is labeled by y_{i} ∈ {−1, +1}, indicating to which of the two classes C_{−} and C_{+} it belongs.

AdaBoost is an iterative algorithm. In the t-th iteration, each of the N training samples is classified into one of the two classes by a weak classifier h_{t}(x_{n}) ∈ {−1, +1}, which can be considered as a hypothesis. The classifier is weak in the sense that its performance only needs to be better than chance, i.e., the error rate is less than 50%. If a sample x_{n} is correctly classified, h_{t}(x_{n}) = y_{n} = ±1, i.e., y_{n} h_{t}(x_{n}) = 1; if it is misclassified, h_{t}(x_{n}) = −y_{n} = ±1, i.e., y_{n} h_{t}(x_{n}) = −1. We need to find the best weak classifier that minimizes the weighted error rate defined as:

ε_{t} = \frac{∑_{y_{n} h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}  (12.41)

where w_{t}(n) is the weight of the nth training samples at the t-th iteration. We can also get the correct rate:

1 − ε_{t} = 1 − \frac{∑_{y_{n} h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)} = \frac{∑_{y_{n} h_{t}(x_{n})=1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}  (12.42)

At the initial stage with t = 0, all N samples are equally weighted by w_{0}(1) = ⋯ = w_{0}(N) = 1, and the weighted error

ε_{0} = \frac{1}{N} ∑_{y_{n} h_{0}(x_{n})=−1} w_{0}(n) = \frac{number of misclassified samples}{total number of samples}  (12.43)

is actually the probability for any sample x in the training set to be misclassified by h_{0}. For the weak classifier the error rate defined above only needs to be lower than 50 percent, i.e., ε_{t} < 1/2. When t > 0, this weight w_{t}(n) will be modified based on whether x_{n} is classified correctly in the subsequent iterations, as we will see below.

At each iteration, a strong or boosted classifier H_{t}(x_{n}) is also constructed based on the linear combination (boosting) of all previous weak classifiers h_{1}(x_{n}), ⋯, h_{t}(x_{n}) for all n = 1, ⋯, N:

F_{t}(x_{n}) = α_{t}h_{t}(x_{n}) + F_{t−1}(x_{n}) = α_{t}h_{t}(x_{n}) + α_{t−1}h_{t−1}(x_{n}) + F_{t−2}(x_{n})
= ⋯ = ∑_{i=1}^{t} α_{i}h_{i}(x_{n})  (12.44)

where the coefficient α_{i} for the weak classifier h_{i}(x_{n}) in the i-th iteration is obtained as discussed below. Taking the sign function of F_{t}(x) we get the strong classifier:

H_{t}(x_{n}) = sign[F_{t}(x_{n})] = \{ \begin{matrix}+1 & F_{t}(x_{n}) > 0 \\ −1 & F_{t}(x_{n}) < 0 \end{matrix}  (12.45)

which classifies a training sample x_{n} into one of the two classes, while the magnitude |F_{t}(x_{n})| represents the confidence of the decision.

The weight w_{t}(n) of x_{n} in the t-th iteration (t > 0) is defined as

w_{t}(n) = e^{−y_{n}F_{t−1}(x_{n})} \{ \begin{matrix}> 1 & if y_{n} F_{t−1}(x_{n}) < 0 (misclassification) \\ < 1 & if y_{n} F_{t−1}(x_{n}) > 0 (correct classification) \end{matrix}

(12.46)

The weight w_{t}(n) of the t-th iteration can be obtained recursively from w_{t−1}(n) of the previous iteration:

w_{t}(n) = e^{−y_{n} F_{t−1}(x_{n})} = e^{−y_{n} [α_{t−1}h_{t−1}(x_{n})+F_{t−2}(x_{n})]}
= e^{−α_{t−1} y_{n}h_{t−1}(x_{n})} e^{−y_{n} F_{t−2}(x_{n})} = e^{−α_{t−1} y_{n}h_{t−1}(x_{n})} w_{t−1}(n)(12.47)

and this recursion can be carried out all the way back to the first iteration with w_{1}(k) = e^{−α_{0} y_{n}h_{0}(x_{n})}w_{0}(n).

The performance of the strong classifier can be measured by the exponential loss, defined as the sume of all N weights:

E_{t+1} = ∑_{n=1}^{N} w_{t+1}(n) = ∑_{n=1}^{N} e^{−y_{n} F_{t}(x_{n})} = ∑_{n=1}^{N} e^{−α_{t} y_{n} h_{t}(x_{n})} w_{t}(n) (t > 1)

(12.48)

The coefficient α_{t} in Eq. (12.44) can now be found as the one that minimizes the exponential loss E_{t+1}. To do so, we first separate the terms in the summation for E_{t+1} into two parts, corresponding to the samples that are classified by h_{t} correctly and incorrectly:

E_{t+1} = ∑_{n=1}^{N} w_{t}(n)e^{−α_{t} y_{n} h_{t}(x_{n})} = ∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)e^{α_{t}} + ∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)e^{−α_{t}}

(12.49)

and then set its derivative with respect to α_{t} to zero:

\frac{dE_{t+1}}{dα_{t}} = ∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)e^{α_{t}} − ∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)e^{−α_{t}} = 0  (12.50)

Solving this equation we get α_{t} as the optimal coefficient that minimizes E_{t+1}:

α_{t} = \frac{1}{2} ln (\frac{∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)}{∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)}) = ln \sqrt{\frac{1 − ε_{t}}{ε_{t}}} > 0  (12.51)

which is greater than zero becauses ε_{t} < 1/2, 1 − ε_{t} > 1/2, and

e^{α_{t}} = \sqrt{\frac{1 − ε_{t}}{ε_{t}}} > 1, e^{−α_{t}} = \sqrt{\frac{ε_{t}}{1 − ε_{t}}} < 1  (12.52)

We now see that if a weak classifier h_{t} has a small error ε_{t}, it will be weighted by a large α_{t} and therefore contributes more to the strong classifier H_{t}; on the other hand, if h_{t} has a large error ε_{t}, it will be weighted by a small α_{t} and contributes less to H_{t}. As the strong classifier H_{t} takes advantage of all previous weak classifiers h_{1}, ⋯, h_{t}, each of which may be more effective in a certain region of the N-D feature space than others, H_{t} can be expected to be a much more accurate classifier than any of the weak classifiers.

Replacing t by t + 1 in Eq. (12.47), we now get

w_{t+1}(n) = w_{t}(n) e^{−α_{t} y_{n}h_{t}(x_{n})} = \{ \begin{matrix}w_{t}(n)\sqrt{\frac{ε_{t}}{1−ε_{t}}} < w_{t}(n) & if y_{n}h_{t}(x_{n}) = 1 \\ w_{t}(n)\sqrt{\frac{1−ε_{t}}{ε_{t}}} > w_{t}(n) & if y_{n}h_{t}(x_{n}) = −1 \end{matrix}

(12.53)

We see that if y_{n}h_{t}(x_{n}) = 1, i.e., x_{n} is classified correctly by h_{t}, it will be weighted more lightly by w_{t+1}(n) < w_{t}(n) in the next iteration, but if y_{n}h_{t}(x_{n}) = −1, i.e., x_{n} is classified incorrectly by h_{t}, it will be weighted more heavily by w_{t+1}(k) > w_{t}(k) in the next iteration, thereby it will be emphasized and have a better chance to be corrected to have more accurately classified by h_{t+1} in the next iteration.

We further consider the ratio of the exponential losses of two consecutive iterations:

\frac{E_{t+1}}{E_{t}} = \frac{∑_{n=1}^{N} w_{t+1}(n)}{∑_{n=1}^{N} w_{t}(n)} = \frac{∑_{n=1}^{N} w_{t}(n)e^{−α_{t} y_{n} h_{t}(x_{n})}}{∑_{n=1}^{N} w_{t}(k)}
= \frac{∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}e^{α_{t}} + \frac{∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}e^{−α_{t}}
= ε_{t} \sqrt{\frac{1 − ε_{t}}{ε_{t}}} + (1 − ε_{t}) \sqrt{\frac{ε_{t}}{1 − ε_{t}}} = 2\sqrt{ε_{t}(1 − ε_{t})} ≤ 1 (12.54)

This ratio is twice the geometric average \sqrt{ε_{t}(1 − ε_{t})} ≤ 1 of ε_{t} and 1 − ε_{t}, which reaches its maximum when ε_{t} = 1 − ε_{t} = 1/2. However, as ε_{t} < 1/2, the ratio is always smaller than 1. We see that the exponential cost can be approximated as an exponentially decaying function from its initial value E_{0} = ∑_{n=1}^{N} w_{0}(n) = N for some ε < 1/2:

lim_{t→∞} E_{t} ≈ lim_{t→∞} (2\sqrt{ε(1 − ε)})^{t} E_{0} = 0  (12.55)

We can therefore conclude that the exponential loss will always decrease, i.e., the error of AdaBoost will always converge to zero.

The weak classifier used in each iteration is typically implemented as a decision stump, a binary classifier that partitions the N-D feature space into two regions. Specifically, as a simple example, a coordinate descent method can be used by which all training samples are projected onto the ith dimension of the feature space d_{i}:

x_{n} = P_{d_{i}}(x_{n}) = \frac{x_{n}^{T} d_{i}}{||d_{i}||} , (n = 1, ⋯, N)  (12.56)

and then classified into two classes by a threshold T along the direction of d_{i}:

h(x_{n}) = \{ \begin{matrix}+1 & if x_{n} < T \\ −1 & if x_{n} > T \end{matrix}  (12.57)

The optimal decision stump is obtained when the following weighted error is minimized with respect to the threshold T along that direction of d_{i}:

ε_{t} = ∑_{n=1}^{N} w_{t}(n) δ(h_{t}(x_{n}) − y_{n}) = ∑_{h_{t}(x_{n})≠y_{n}} w_{t}(n)  (12.58)

Alternatively, the weak classifier can also be obtained based on the principal component analysis (PCA) by partitioning the N-D feature space along the directions of the eigenvectors of the between-class scatter matrix

S_{b} = \frac{1}{N} [N_{−}(m_{−1} − m)(m_{−1} − m)^{T} + N_{+}(m_{+1} − m)(m_{+1} − m)^{T} ] (12.59)

where N_{−} and N_{+} are the numbers of samples in the two classes (N_{−}+N_{+} = N), and

m_{±} = \frac{1}{N_{±}} ∑_{y_{n}=±1} w(n) x_{n}, m = \frac{1}{N} ∑_{n=1}^{N} w(n) x_{n}  (12.60)

are the weighted mean vectors of the two classes and the total weighted mean vector of all N samples in the training set. The training samples are likely to be better separated along these directions of the eigenvectors of S_{b}, which measures the separability of the two classes. The eigenequations of S_{b} is:

S_{b} = ΦΛΦ^{−1} = ΦΛΦ^{T}  (12.61)

As S_{b} is symmetric, the eigenvector matrix Φ is orthonormal and its columns can be used as an orthogonal basis that spans the feature space as well as the standard basis. The same binary threshold classification considered above for the weak classifiers can be readily applied along these PCA bases.

The Matlab code of the main iteration loop of the algorithm is listed below, followed by the essential functions called by the main loop. Here T is the maximum number of iteration, and N is the total number of training samples.

The algorithm can be carried out in the feature space based on either the PCA basis (columns of the eigenvector matrix Φ), or the original standard basis (columns of the identity matrix I).

The decision stump is implemented by a binary classifier, which partitions all training samples projected onto a 1-D space into two groups by a threshold value, which needs to be optimal in terms of the number of misclassification. A parameter plt is used to indicate the polarity of the binary classification, i.e., which of the two class C_{1} and C_{0} is on the lower (or higher) side of the threshold.

    Plt=[];              % polarity of binary classification
    Tr=[];               % transformation of basis vectors
    Th=[];               % threshold of binary classification
    Alpha=[];            % alpha values
    h=zeros(T,N);        % h functions
    F=zeros(T,N);        % F functions
    w=ones(T,N);         % weights for N training samples
    Er=N;                % error initialized to N
    t=0;                 % iteration index
    while t<T  & Er>0                 % the main iteration
        t=t+1;
        [tr th plt er]=WeakClassifier(X,y,w(t,:),PCA); % weak classifier
        alpha=log(sqrt((1-er)/er));   % update alpha
        Alpha=cat(1,Alpha,alpha);     % record alpha
        Tr=cat(2,Tr,tr');             % record transform vector
        Th=cat(1,Th,th);              % record threshold
        Plt=cat(1,Plt,plt);           % record polarity
        x=tr*X;                       % carry out transform

c=sqrt(er/(1-er));
for n=1:N                     % update weights
    h(t,n)=h_function(x(n),th,plt);     % find h function
   if  h(t,n)*y(n)<0
       w(t+1,n)=w(t,n)/c;     % update weights
   else
       w(t+1,n)=w(t,n)*c;
   end
end
F(t,:)=Alpha(t)*h(t,:);       % get F functions
if t>1
    F(t,:)=F(t,:)+F(t-1,:);
end
Er=sum(sign(F(t,:).*y)==-1);  % error of strong classifier, number of misclassifications
fprintf('%d: %d/%d=%f\n',t,Er,N,Er/N)
end

Here are the functions called by the main iteration loop above:

function [Tr,Th Plt Er]=WeakClassifier(X,y,w,pca)
% X: N columns each for one of the N training samples
% y: labeling of X
% w: weights for N training samples
% pca: use PCA dimension if pca~=0
% Er: minimum error among all D dimensions
% Tr: transform vector (standard or PCA basis)
% Th: threshold value
% Plt: polarity
[D N]=size(X);
n0=sum(y>0);            % number of samples in class C+
n1=sum(y<0);            % number of samples in class C-
if pca                  % find PCA basis
for i=1:D
    Y(i,:)=w.*X(i,:);
end
X0=Y(:,find(y>0));  % all samples in class C+
X1=Y(:,find(y<0));  % all samples in class C-
m0=mean(X0')';      % mean of C+
m1=mean(X1')';      % mean of C-
mu=(n0*m0+n1*m1)/N; % over all mean
Sb=(n0*(m0-mu)*(m0-mu)'+n1*(m1-mu)*(m1-mu)')/N; % between-class scatter matrix
[v d]=eig(Sb);      % eigenvector and eigenvalue matrices of Sb
else
v=eye(N);           % standard basis
end

Er=9e9;
for i=1:D               % for all D dimensions
tr=v(:,i)';         % get transform vector from identity or PCA matrix
x=tr*X;             % rotate the vector
[th plt er]=BinaryClassifier(x,y,w); % binary classify N samples in 1-D
er=0;
for n=1:N
    h(n)=h_function(x(n),th,plt);  % h-function of nth sample
    if h(n)*y(n)<0                 % if misclassified
        er=er+w(n);                % add error
    end
end
er=er/sum(w);                      % total error of dimension d
if Er>er                           % record info corresponding to min error
    Er=er;                         % min error
    Plt=plt;                       % polarity
    Th=th;                         % threshold
    Tr=tr;                         % transform vector
end
end
end
function h=h_function(x, th, plt)
if xor(x>th, plt)
h=1;
else
h=-1;
end
end
function [Th Plt Er]=BinaryClassifier(x,y,w)
N=length(x);
[x1 i]=sort(x);            % sort 1-D data x
y1=y(i);                   % reorder the targets
w1=w(i);                   % reorder the weights
Er=9e9;
for n=1:N-1                % for N-1 ways of binary classification
e0=sum(w1(find(y1(1:n)==1)))+sum(w1(n+find(y1(n+1:N)==-1)));
e1=sum(w1(find(y1(1:n)~=1)))+sum(w1(n+find(y1(n+1:N)~=-1)));
if e1 > e0                 % polarity: left -1, right +1
    plt=0; er=e0;
else                       % polarity: left +1, right -1
    plt=1; er=e1;
end

        if Er > er                 % update minimum error for kth dimension
            Er=er;                 % minimum error
            Plt=plt;               % polarity
            Th=(x1(k)+x1(k+1))/2;  % threshold
        end
    end
end

Example 0 This example shows the classification of an XOR data set (used previously the test the naive Bayes mehtod), containing two classes of N = 200 simples each in 2-D space, represented by the red and blue dots in the figure below. The intermediate results after 4, 8, 16 and 32 iterations are shown in to illustrate the progress of the iteration, when the error rate continuously reduces from 134/400, 85/400, 81/400, 58/100.

After over 350 iterations the error rate eventually reduced to zero (guaranteed by the AdaBoost method). The sequence of binary participation Although all training samples are correctly classified eventually, the result suffers the problem of overfitting.

Example 1 This example shows the classification of two classes of N = 500 simples in 2-D space, which is partitioned along both the standard basis vectors (left) and the PCA directions (right). The PCA method performances significantly better in this example as its error reduces faster and it converges to zero after 61 iterations, while the error of the method based on the standard axes (in vertical and horizontal directions) does not go down to zero even after 120 iterations. This faster reduction of error of the PCA method can be easily understood as many different directions are used to partition the space into two regions, while in the coordinate descent method the partitioning of the space is limited to either of the two directions.

How the AdaBoost is iteratively trained can be seen in the figure below. First, the error rate, the ratio between the number of misclassified samples and the total number of samples, for both the training and testing samples, are plotted as the iteration progresses (top). Also, the weighted error ε_{t}, the exponential costs E_{t}, and the ratio E_{t+1}/E_{t} = 2\sqrt{ε_{t}(1 − ε_{t})} are all plotted (bottom). We see that ε_{t} < 0.5 and E_{t+1}/E_{t} < 1 for all steps, and the exponential cost E_{t} attenuates exponentially towards zero.

Example 2 In this dataset, 100 training samples of two classes (50 each) forms four clusters arranged in the 2-D space as an XOR pattern as shown figure (top-left), and 100 testing samples of the same distribution are classified by the AbaBoost algorithm trained by both the coordinate descent and PCA methods.

We see that the coordinate descent method performs very poorly in both the slow convergence (more than 400 iterations) during training and high classification error rate (more than 1/3) during testing. The partitioning of the 2-D space is an obvious over fitting of the training set instead of reflecting the actual distribution of the two classes (four clusters). On the other hand, the PCA method converges quickly (10 iterations) and classifies the testing samples with very low error rate. The space is clearly partitioned into two regions corresponding to the two classes.

13 Support Vector machine

13.1 Maximal Margin and Support Vectors

The support vector machine (SVM) is a supervised binary classifier trained by a dsta set {(x_{n}, y_{n}), n = 1, ⋯, N} of which data sample x_{n}, a vector in the d-dimensional feature space, belongs to either class C_{+} if labeled by y_{n} = 1 or class C_{−} if labeled by y_{n} = −1. The result of the training process is the decision boundaries or decision plane H_{0} in the feature space, described by the linear decision equation:

f(x) = w^{T} x + b = ∑_{i=1}^{d} x_{i}w_{i} + b = 0  (13.1)

in terms of its normal vector w and intercept b. Once the two parameters w and b are determined based on the training set, any unlabeled sample x can be classified into either of the two classes:

if f(x) = w^{T} x + b \{ \begin{matrix}> 0 \\ = 0 \\ < 0 \end{matrix} , then \{ \begin{matrix}x ∈ C_{+} \\ x ∈ H_{0} \\ x ∈ C_{−} \end{matrix}  (13.2)

The initial setup of the SVM algorithm seems the same as the method of linear regression as a linear binary classifier when the linear regression function f(x) = w^{T} x + b is thresholded by zero to become f(x) = 0. But here for the SVM, the parameters w and b are determined in such a way that the corresponding decision plane separates the training samples belonging to the two classes optimally, so that its distance to the closest samples on either side, called support vectors, is maximized. Also, as kernel mapping can be applied to the SVM, it can also be used for nonlinear classification.

We rewrite the decision equation f(x) = w^{T} x + b = 0 as

p_{w}(x) = \frac{w^{T} x}{||w||} = −\frac{b}{||w||}  (13.3)

and realize this is the projection of any point x ∈ H_{0} on the decision plane H_{0} onto its normal direction w, of which the absolute value is the distance between H_{0} and the origin:

d(H_{0}, 0) = |p_{w}(x)| = \frac{|b|}{||w||}  (13.4)

We further find the projection of any point x′ ∉ H_{0} off the decision plane H_{0} onto w as p_{w}(x′) = w^{T} x′/||w||, and its distance to H_{0} as:

d(H_{0}, x′) = |p_{w}(x′) − p_{w}(x)| = |\frac{w^{T} x′}{||w||} − (−\frac{b}{||w||}) | = \frac{|w^{T} x′ + b|}{||w||} = \frac{|f(x′)|}{||w||}

(13.5)

We desire to find the optimal decision plane H_{0} that separates the training samples belonging to the two different classes, assumed to be linearly separable, in such a way that its distance to the support vectors, denoted by x_{sv}, is maximized:

d(H_{0}, x_{sv} ∈ C_{+}) = d(H_{0}, x_{sv} ∈ C_{−}) = d(H_{0}, x_{sv}) = \frac{|f(x_{sv})|}{||w||}  (13.6)

We further consider two additional planes H_{+} and H_{−} that are in parallel with H_{0} and pass through the support vectors x_{sv} on either side of H_{0}:

f_{±}(x) = f(x) ± c = w^{T} x + b ± c = w^{T} x + b ± 1 = 0  (13.7)

As these equations can be arbitrarily scaled, we can let c = 1 for convenience. Based on Eq. (13.4), the distances from H_{+} or H_{−} to the origin can be written as

d(H_{±}, 0) = \frac{|b ± c|}{||w||} = \frac{|b ± 1|}{||w||}  (13.8)

and their distances to the decision plane H_{0}, called the decision margin, can be found as:

d(H_{±}, H_{0}) = |d(H_{±}, 0) − d(H_{0}, 0)| = |\frac{|b ± c|}{||w||} − \frac{|b|}{||w||}| = \frac{|c|}{||w||} = \frac{1}{||w||} (13.9)

To maximize this margin, we need to minimize ||w||.

For these planes to correctly separate all samples in the training set {(x_{n}, y_{n}) (n = 1, ⋯, N)}, they have to satisfy the following two conditions:

\{ \begin{matrix}f_{−}(x_{n}) = w^{T} x_{n} + b − 1 ≥ 0 & if y_{n} = 1 \\ f_{+}(x_{n}) = w^{T} x_{n} + b + 1 ≤ 0 & if y_{n} = −1 \end{matrix}  (n = 1, ⋯, N) (13.10)

where the equality is satisfied by the support vectors on H_{+} or H_{−}, while the inequalities are satisfied by all other samples farther away from H_{0} behind H_{+} or H_{−}. The two conditions above can be combined to become:

y_{n}(w^{T} x_{n} + b) ≥ 1, (n = 1, ⋯, N)  (13.11)

Now the task of finding the optimal decision plane H_{0} can be formulated as a constrained minimization problem:

minimize: \frac{1}{2}||w||^{2} = \frac{1}{2}w^{T} w
subject to: y_{n}(x_{n}^{T} w + b) ≥ 1, (n = 1, ⋯, N)  (13.12)

Example:

The straight line in 2D space shown above, denoted by H_{0}, is described by the following linear equation

f(x) = w^{T} x + b = [w_{1}, w_{2}] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] + b = [1, 2] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] − 1 = x_{1} + 2x_{2} − 1 = 0

(13.13)

The distance from H_{0} to the origin is:

d(H_{0}, 0) = \frac{|b|}{||w||} = \frac{1}{\sqrt{w_{1}^{2} + w_{2}^{2}}} = \frac{1}{\sqrt{1^{2} + 2^{2}}} = \frac{1}{\sqrt{5}} = 0.447  (13.14)

which is invariant with respect to any scaling of the equation. Consider three points in the space:

• x_{0} = [0.5, 0.25]^{T}, f(x_{0}) = 0.5 + 2 × 0.25 − 1 = 0, i.e., x_{0} is on the plane. Its distance to the plane is d(x_{0}, H) = f(x_{0})/||w|| = 0.

• x_{1} = [1, 0.25]^{T}, f(x_{1}) = 1 + 2 × 0.25 − 1 = 0.5 > 0, i.e., x_{1} is above the straight line, its distance to the plane is d(x_{1}, H) = f(x_{1})/||w|| = |0.5|/\sqrt{5} = 0.2235.

• x_{2} = [0.5, 0]^{T}, f(x_{2}) = 0.5+2×0−1 = −0.5 < 0, i.e., x_{2} is below the straight line, its distance to the plane is d(x_{2}, H) = f(x_{2})/||w|| = | − 0.5|/\sqrt{5} = 0.2235.

These two points x_{1} = [1, 0.25]^{T} and x_{2} = [0.5, 0]^{T} with equal distance to H_{0}, the straight line f(x) = x_{1} + 2x_{2} − 1 = 0, can be assumed to be two support vectors x_{sv} on either side of H_{0}, and

|f(x_{sv})| = |w^{T} x_{1} + b| = |w^{T} x_{2} + b| = 0.5  (13.15)

Multiplying 2 on both sides of the decision equation f(x) = 0, it is scaled to become

f(x) = w^{T} x + b = 2x_{1} + 4x_{2} − 2 = 0  (13.16)

and the two equations diescribing H_{+} parallel to H_{0} and passing through x_{1} and H_{−} passing through x_{2} are

H_{+} : 2x_{1} + 4x_{2} − 2 − 1 = 2x_{1} + 4x_{2} − 3 = 0
H_{−} : 2x_{1} + 4x_{2} − 2 + 1 = 2x_{1} + 4x_{2} − 1 = 0

Their distances to the origin are

d(H_{+}, 0) = \frac{| − 3|}{||w||} = 1.341, d(H_{−}, 0) = \frac{| − 1|}{||w||} = 0.447  (13.17)

and the distance between H_{−} and H_{+} is indeed 2/||w||:

d(H_{−}, H_{+}) = d(H_{+}, 0) − d(H_{−}, 0) = \frac{2}{||w||} = 0.894  (13.18)

twice the distance between H_{0} and the origin d(H_{0}, 0) = 0.447 found previously. For reasons to be discussed later, instead of directly solving the constrained minimization problem in Eq. (13.12), now called the primal problem, we actually solve the dual problem (Section 2.2.3). Specifically, we first construct the Lagrangian function of the primal problem:

L_{p}(w, b, α) = \frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))  (13.19)

where α_{1}, ⋯, α_{N} are the Lagrange multipliers, which is called the primal function. Here for this minimization problem with non-positive constraints, the Lagrangian multipliers are required to be negative, α_{n}, according to Table 2.167 in Section 2.2.2, if a minus sign is used for the second term. However, to be consistent with most SVM literatures, we use the positive sign for the second term and require α_{n} ≥ 0. Note that if x_{n} is a support vector on either H_{+} or H_{−}, i.e., the equality constraint y_{n}(w^{T} x_{n} + b) = 1 holds, then α_{n} > 0; but if x_{n} is not a support vector, the equality constraint does not hold, and α_{n} = 0.

We next find the minimum (or infimum) as the lower bound of the primal function L_{p}(w, b) in Eq. (13.19), by setting to zero its partial derivatives with respect to both b and w:

\frac{∂}{∂b}L_{p}(w, b) = \frac{∂}{∂b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))] = ∑_{n=1}^{N} α_{n}y_{n} = 0

(13.20)

and

\frac{∂}{∂w} L_{p}(w, b) = \frac{∂}{∂w} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))] = w−∑_{n=1}^{N} α_{n}y_{n}x_{n} = 0,

(13.21)

i.e.,

w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}  (13.22)

Substituting these back into the primal function L_{p}(w, b, α), we get its lower bound as a function of the Lagrange multipliers {α_{1}, ⋯, α_{N}}, called the dual function:

L_{d}(α) = inf_{w,b} L_{p}(w, b, α) = inf_{w,b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))]

= inf_{w,b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n} − w^{T} ∑_{n=1}^{N} α_{n}y_{n}x_{n} − b ∑_{n=1}^{N} α_{n}y_{n}]

= \frac{1}{2} ( ∑_{m=1}^{N} α_{m}y_{m}x_{m})^{T} (∑_{n=1}^{N} α_{n}y_{n}x_{n}) + ∑_{n=1}^{N} α_{n} − ( ∑_{m=1}^{N} α_{m}y_{m}x_{m})^{T} (∑_{n=1}^{N} α_{n}y_{n}x_{n})

= ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m})  (13.23)

To further find the greatest or tightest lower bound of the primal function, we need to maximize this dual function L_{d}(α) with respect to the Lagrange multipliers, subject to the constraint imposed by Eq. (13.20):

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m}) = 1^{T} α − \frac{1}{2}α^{T} Qα

subject to: ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N)  (13.24)

where Q is an N by N symmetric matrix of which the component in the mth row and nth column is Q(m, n) = y_{m}y_{n}x_{m}^{T}x_{n} (m, n = 1, ⋯, N). Now we have converted the primal problem of constrained minimization of the primal function L_{p}(w, b, α) with respect to w and b to its dual problem of linearly constrained maximization of the dual function L_{d}(α) with respect to {α_{1}, ⋯, α_{N}}. Solving this quadratic programming (QP) problem (Section 2.2.6), we get all Lagrange multipliers α_{n} ≥ 0, (n = 1, ⋯, N). All training samples x_{n} corresponding to positive Lagrange multipliers α_{n} > 0 are support vectors, while others corresponding to α_{n} = 0 are not support vectors.

We can now find the normal direction w of the optimal decision plane based on Eq. (13.22):

w = ∑_{n=1}^{N} α_{n}y_{n}x_{n} = ∑_{n∈sv} α_{n}y_{n}x_{n} = ∑_{x_{n}∈C_{+}, n∈sv} α_{n}x_{n} − ∑_{x_{n}∈C_{−}, n∈sv} α_{n}x_{n}

(13.25)

Note that the normal vector w is the difference between the weighted means of the support vectors in class C_{+} (first term) and class C_{−} (second term), and it is determined only by the support vectors.

Having found {α_{1}, ⋯, α_{N}}, we can also find b based on the equality of Eq. (13.11) for any of the support vectors:

y_{n} (w^{T} x_{n} + b) = 1 or w^{T} x_{n} + b = y_{n}, (n ∈ sv)  (13.26)

(recall y_{n}^{2} = 1). Solving the equation for b we get:

b = y_{n} − w^{T} x_{n} = y_{n} − ∑_{m∈sv} α_{m}y_{m}(x_{m}^{T}x_{n}), (n ∈ sv)  (13.27)

All support vectors should yield the same result. Computationally we simply get b as the average of the above for all support vectors.

We note that both w and b depend only on the support vectors on planes H_{+} and H_{−}, corresponding to a positive α_{n} > 0, while all remaining samples corresponding to α_{n} = 0 are farther away from the decision plane H_{0}, behind either H_{+} or H_{−}. We see that the SVM is solely based on those support vectors in the training set, once they are identified during the training process, all other non-support vectors are irrelevant.

Having obtained w and b as shown above as the training process, any unlabeled sample x can be classified into either of the two classes depending on whether the following decision function is greater or smaller than zero:

f(x) = w^{T} x+b = ( ∑_{n∈sv} α_{n}y_{n}x_{n})^{T} x+b = ∑_{n∈sv} α_{n}y_{n} (x_{n}^{T} x)+b \{ \begin{matrix}> 0 & x ∈ C_{+} \\ < 0 & x ∈ C_{−} \end{matrix}

(13.28)

We also get:

||w||^{2} = w^{T} w = ( ∑_{n∈sv} α_{n}y_{n}x_{n}^{T} ) ( ∑_{m∈sv} α_{m}y_{m}x_{m}) = ∑_{n∈sv} α_{n}y_{n} ∑_{m∈sv} α_{m}y_{m}(x_{n}^{T} x_{m})
= ∑_{n∈sv} α_{n}y_{n}(y_{n} − b) = ∑_{n∈sv} α_{n}(1 − y_{n}b) = ∑_{n∈sv} α_{n} − b ∑_{n∈sv} α_{n}y_{n} = ∑_{n∈sv} α_{n} (13.29)

The last equality is due to equality constraints of the dual problem ∑_{n=1}^{N} α_{n}y_{n} = 0. Now the decision margin can be written as:

\frac{1}{||w||} = ( ∑_{n∈sv} α_{n})^{−1/2}  (13.30)

The SVM algorithm for binary classification can now by summarized as the following steps:

• Find the Lagrange multipliers {α_{1}, ⋯, α_{N}} by solving the QP problem in Eq. (13.24);

• Find the normal vector w by Eq. (13.25);

• Find the bias b by Eq. (13.27);

• Classify unlabeled x by Eq. (13.28).

As the last expression of the decision function in Eq. (13.28) depends only on α_{n} as well as x_{n} and y_{n} in the training set, the normal vector w is never needed, and the second step above for calculating w by Eq. (13.25) can be dropped.

We prefer to solve this dual problem not only because it is easier than the original primal problem, but also, more importantly, for the reason that all data points appear only in the form of an inner product x_{n}^{T} x_{m} in the dual problem, allowing the kernel method to be used, as discussed later.

The Matlab code for the essential part of the algorithm is listed below, where X and y are respectively the data array composed of N training vectors [x_{1}, ⋯, x_{N}] and their corresponding labelings y_{1}, ⋯, y_{N}, and IPmethod is a function that implements the interior point method (Section 2.2.7) for solving a general QP problem of minimizing x^{T} Qx/2 + c^{T} x subject to the linear equality constraints Ax = b and x ≥ 0.

    [X y]=getData;               % get the training data
    [m n]=size(X);
    Q=(y'*y).*(X'*X);            % compute the quadratic matrix
    c=-ones(n,1);                % coefficients of linear term

    A=y;                         % coefficient matrix and
    b=0;                         % constants of linear equality constraints
    alpha=0.1*ones(n,1);         % initial guess of solution
    [alpha mu lambda]=IPmethod(Q,A,c,b,alpha);  % solve QP to find alphas
                                                % by interior point method
    I=find(abs(alpha)>10^(-3));  % indecies of non-zero alphas
    asv=alpha(I);                % non-zero alphas
    Xsv=X(:,I);                  % support vectors
    ysv=y(:,I);                  % their labels
    w=sum(repmat(ysv.*asv,m,1).*Xsv,2);  % normal vector (not needed)
    bias=mean(ysv-w'*Xsv);               % bias

Example: The training set contains two classess of 2-D points with Gaussian distributions generated based on the following mean vectors and covariance matrices:

m_{−} = [ \begin{matrix}3.5 \\ 0 \end{matrix} ] , S_{−} = [ \begin{matrix}1 & 0.5 \\ 0.5 & 1 \end{matrix} ] , m_{+} = [ \begin{matrix}0 \\ 3.5 \end{matrix} ] , S_{+} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ]

(13.31)

The SVM algorithm finds the weight vector w and constant b of the optimal boundary between the two classes based on three support vectors, all listed below. Note that decision boundary is completely dictated by the three support vectors and the margion distance between them is maximized.

\begin{matrix}n & α_{n} & x_{n} = [x_{1}, x_{2}] & y_{n} \\ 40 & 0.52 & x = [4.43, 2.44] & −1 \\ 52 & 3.11 & x = [1.17, 0.74] & −1 \\ 103 & 3.64 & x = [1.30, 1.64] & 1 \end{matrix} ,  w = [ \begin{matrix}−1.25 \\ 2.39 \end{matrix} ] ,  b = −1.31

(13.32)

13.2 Kernel Mapping

The SVM algorithm above converges only if the data points of the two classes in the training set are linearly separable. If this is not the case, we can consider the following kernel method.

In the kernel method, all data points as a vector x = [x_{1}, ⋯, x_{d}]^{T} in the original d-dimensional feature space are mapped by a kernel function z = ϕ(x) into a higher, possibly infinite, dimensional space

x ⟹ z = ϕ(x)  (13.33)

The presumption of the method is that the data points only appear in the form of inner product x_{m}^{T}x_{n} in the algorithm, and the kernel function z = ϕ(x) never needs to be actually carried out. In fact, the form of the kernel function ϕ(x) and the dimensionality of the higher dimensional space do not need to be explicitly specified or known.

The motivation for such a kernel mapping is that the relevant operations such as classification and clustering may be carried out much more effectively once the dataset is mapped to the higher dimensional space. For example, classes not linearly separable in the original d-dimensional feature space may be trivially separable in a higher dimensional space, as illustrated by the following examples.

Example 1

In 1-D space, classes C_{−} = {x|(a ≤ x ≤ b)} and C_{+} = {x|(x ≤ a) or (x ≥ b)} are not linearly separable. By the following mappping from 1-D space to 2-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \end{matrix} ] = [ \begin{matrix}x \\ (x − (a + b)/2)^{2} \end{matrix} ]  (13.34)

the two classes can be separated by a threshold in the second dimension of the 2-D space.

Example 2:

The method above can be generalized to higher dimensional spaces such as mapping from 2-D to 3-D space. Consider two classes in a 2-D space that are not linearly separable: C_{−} = {x, ||x|| < D} and C_{+} = {x, ||x|| > D}. However, by the following mappping from 2-D space to 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}^{2} + x_{2}^{2} \end{matrix} ]  (13.35)

the two classes can be trivially separated linearly by thresholding in the third dimension of the 3-D space.

Example 3:

In 2-D space, in the exclusive OR data set, the two classes of C_{−} containing points in quadrants I and III, and C_{+} containing points in quadrants II and IV are not linearly separable. However, by mapping the data points to a 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}x_{2} \end{matrix} ]  (13.36)

the two classes can be separated by simply thresholding in the third dimension of the 3-D space.

Definition: A kernel is a function that takes two vectors x_{m} and x_{n} as arguments and returns the inner product of their images z_{m} = ϕ(x_{m}) and z_{n} = ϕ(x_{n}):

K(x_{m}, x_{n}) = ϕ(x_{m})^{T} ϕ(x_{n}) = z_{m}^{T}z_{n}  (13.37)

The kernel function takes as input some two vectors x_{m} and x_{n} in the original feature space, and returns a scalor value as the inner product z_{m} and z_{n} in some higher dimensional space. If the data points in the original space only appear in the form of inner product, then the kernel function z = ϕ(x), called the kernel-induced implicit mapping, never needs to be explicitly specified, and the dimension of the new does not even need to be known.

The following is a set of commonly used kernel functions K(x, x′), which can be represented as an inner product of two vectors z = ϕ(x) and z′ = ϕ(x′) in a higher dimensional space.

• linear kernel (no kernel mapping)

Assume x = [x_{1}, ⋯, x_{d}]^{T}, x′ = [x′_{1}, ⋯, x′_{d}]^{T},

K(x, x′) = x^{T} x′ = ∑_{i=1}^{d} x_{i}x′_{i}  (13.38)

• polynomial kernels The binomial theorem states:

(x + y)^{n} = ∑_{k=0}^{n} ( \begin{matrix}n \\ k \end{matrix} ) x^{n−k}y^{k} = ∑_{k=0}^{n} \frac{n!}{k!(n − k)!} x^{n−k}y^{k}  (13.39)

where the binomial coefficient

( \begin{matrix}n \\ k \end{matrix} ) = \frac{n!}{k!(n − k)!}  (13.40)

is the number of ways to distribute n items into two bins (k in one and n−k in the other). This result can be generalized to the multinormial case:

(x_{1}+⋯+x_{d})^{n} = ∑ ; ∑_{i=1}^{d} k_{i}=n ( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}} = ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!}x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}

(13.41)

where the multinomial coefficient

( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) = \frac{n!}{k_{1}! ⋯ k_{d}!}  (13.42)

is the number of ways to distribute n balls into d bins with k_{i} balls in the ith

bin (https://www.statlect.com/mathematical-tools/partitions), and

the summation is over all possible ways to get d non-negative integers k_{1}, ⋯, k_{d} that add up to n.

Now consider the homogeneous polynomial kernel for d-dimensional vectors x = [x_{1}, ⋯, x_{d}]^{T} defined as

K(x, x′) = (x^{T} x′)^{n} = (x_{1}x′_{1} + ⋯ + x_{d}x′_{d})^{n}
= ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}}) = ϕ(x)^{T} ϕ(x′) = z^{T} z′ (13.43)

where

z = ϕ(x) = [\sqrt{\frac{n!}{k_{1}! ⋯ k_{d}!}} (x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}) , (k_{i} ≥ 0, ∑_{i=1}^{d} k_{i} = n)]^{T} (13.44)

In particular, when d = 2 and n = 2, the polynomial kernel defined over 2-D vectors x = [x_{1}, x_{2}]^{T} is:

K(x, x′) = (x^{T} x′)^{2} = (x_{1}x′_{1}+x_{2}x′_{2})^{2} = (x_{1}x′_{1})^{2}+2x_{1}x′_{1}x_{2}x′_{2}+(x_{2}x′_{2})^{2} = ϕ(x)^{T} ϕ(x′) = z^{T} z′

(13.45)

where z = ϕ(x) = [x_{1}^{2}, \sqrt{2}x_{1}x_{2}, x_{2}^{2}] is a mapping from x in 2-D space to z in 3-D space.

A non-homogeneous polynomial kernel is defined as

K(x, x′) = (1 + x^{T} x′)^{n}  (13.46)

• The radial basis function (RBF) kernel The RBF kernel is defined as

K(x, x′) = e^{−||x−x′||^{2}/2σ^{2}} = e^{−γ||x−x′||^{2}}  (13.47)

where γ = 1/2σ^{2} is a parameter that can be adjusted to fit each specific dataset. This kernel can be wriiten as the inner product of two infinite dimensional vectors (for simplicity, we assume σ = 1):

K(x, x′) = e^{−||x−x′||^{2}/2} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} e^{x^{T} x′} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} \frac{(x^{T} x′)^{n}}{n!}

= e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} [\frac{1}{n!} ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}})]

= ∑_{n=0}^{∞} ∑ ; ∑_{i=1}^{d} k_{i}=n (e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}) (e^{−||x′||^{2}/2} \frac{x_{1}^{′k_{1}} ⋯ x_{d}^{′k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}})
= ϕ(x)^{T} ϕ(x′) = z^{T} z′  (13.48)

where

z = ϕ(x) = [e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}, (n = 0, ⋯, ∞, ∑_{k=1}^{n} k_{i} = n)]^{T}

(13.49)

is a vector in an infinite dimensional space. In particular, when d = 1 we have

K(x, x′) = e^{−(x−x′)^{2}/2} = e^{−x^{2}/2} e^{−x^{′2}/2} e^{xx′} = e^{−x^{2}/2} e^{−x^{′2}/2} ∑_{n=0}^{∞} \frac{(xx′)^{n}}{n!}

= ∑_{n=0}^{∞}(e^{−x^{2}/2} x^{n}/\sqrt{n!}) (e^{−x^{′2}/2} x^{′n}/\sqrt{n!})  (13.50)

where z = ϕ(x) = [e^{−x^{2}/2} x^{n}/\sqrt{n!}, (n = 0, ⋯, ∞)]^{T} is a kernel function that maps a 1-D space into an infinite dimensional space.

The method of kernel mapping can be applied to the SVM algorithm consider previously as all data points appear in the algorithm are in the form of an inner product. Specifically, during the training process, we replace the inner product x_{m}^{T}x_{n} in both Eq. (13.24) and Eq. (13.27) by the kernel functioin K(x_{m}, x_{n}) to get

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m}K(x_{n}, x_{m})

subject to: ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N) (13.51)

and

b = y_{n} − ∑_{m∈sv} α_{m}y_{m}K(x_{m}, x_{n}), (n ∈ sv)  (13.52)

We also replace the inner product x_{n}^{T} x in Eq. (13.28) by K(x_{n}, x) for the classification of any unlabeled point z = ϕ(x):

f(z) = w^{T} z+b = ∑_{n∈sv} α_{n}y_{n} (z_{n}^{T} z)+b = ∑_{n∈sv} α_{n}y_{n} K(x_{n}, x)+b \{ \begin{matrix}> 0 & x ∈ C_{+} \\ < 0 & x ∈ C_{−} \end{matrix}

(13.53)

Again, we note that the normal vector w in Eq. (13.25) never needs to be explicitely calculated. As now both the training and classification are carried out in some higher dimensional space z = ϕ(x), in which the classes are more likely linearly separable, the classification can be more effectively. More generally, the kernel method can be applied to any algorithm, so long as the data always appear in the form of an inner product.

13.3 Soft Margin SVM

When the two classes C_{−} and C_{+} are not linearly separable, the condition for the optimal hyperplane in Eq. (13.11) can be relaxed by including an extra error term ξ_{n} ≥ 0:

y_{n}(x_{n}^{T} w + b) ≥ 1 − ξ_{n}, (n = 1, ⋯, N)  (13.54)

For better classification result, this error ξ_{n} needs to be minimized as well as ||w||. Now the primal problem in Eq. (13.12) can be modified to the following minimization problem with an objective function and two sets of inequality (nonnegative) constraints:

minimize: \frac{1}{2}||w||^{2} + C ∑_{n=1}^{N} ξ_{n}
subject to: y_{n}(x_{n}^{T} w + b) + ξ_{n} − 1 ≥ 0, and ξ_{n} ≥ 0; (n = 1, ⋯, N) (13.55)

The value of the parameter C can be adjusted to make a proper trade-off between under fitting emphasizing the global distribution of the training samples, and over fitting emphasizing the local samples close to the decision boundary H_{0}. A large C value emphasizes the minimization of the error term in the objective function, and the resulting H_{0} is mostly dictated by a small number of local support vectors in the region between the class, including possibly some noisy outliers, thereby risking the problem of overfitting. On the other hand, a small C value de-emphasizes the error term and the classifier can tolerate greater errors, thereby allowing more data samples farther away from H_{0} to become support vectors and play a role in determining H_{0}. Consequently the classification trends to better reflect the overall distribution of the data samples, but with the risk of overfitting.

This constrained minimization problem can be converted to the minimization of the corresponding Lagrangian:

L_{p}(w, b, ξ, α, μ) = \frac{1}{2}||w||^{2} + C ∑_{n=1}^{N} ξ_{n} − ∑_{n=1}^{N} α_{n}[y_{n}(w^{T} x + b) + ξ_{n} − 1] − ∑_{n=1}^{N} β_{n}ξ_{n}

(13.56)

where β_{n} as well as α_{n} is the Lagrange multipliers for the two sets of constraints. As they are non-negative (instead of non-positive as in previous section), minus sign is used in front of the last two terms for the constraints, so that the Lagrange multipliers α_{n} and β_{n} are still required to be positive (in consistent with Table 2.167). The KKT conditions (Eqs. (2.189) through (2.192)) of this minimization problem are listed below (for all n = 1, ⋯, N):

• Stationarity:

\frac{∂L_{p}}{∂w} = w − ∑_{n=1}^{N} y_{n}α_{n}x_{n} = 0, i.e. w = ∑_{n=1}^{N} y_{n}α_{n}x_{n}; (13.57)

\frac{∂L_{p}}{∂b} = ∑_{n=1}^{N} y_{n}α_{n} = 0;  (13.58)
\frac{∂L_{p}}{∂ξ_{n}} = C − α_{n} − β_{n} = 0;  (13.59)

• Primal feasibility:

y_{n}(w^{T} x + b) + ξ_{n} − 1 ≥ 0, ξ_{n} ≥ 0,  (13.60)

• Complementarity:

α_{n}[y_{n}(w^{T} x + b) + ξ_{n} − 1] = 0, β_{n}ξ_{n} = 0,  (13.61)

• Dual feasibility:

α_{n} ≥ 0, β_{n} ≥ 0  (13.62)

Depending on the value of α_{n}, there exist three possible cases, due to the complementarity (Eq. (13.61)):

• If α_{n} = 0, then β_{n} = C − α_{n} = C > 0, ξ_{n} = 0, and we have

y_{n}(w^{T} x_{n} + b) − 1 ≥ 0  (13.63)

• If 0 < α_{n} < C, i.e., α_{n} ≠ 0, then we have

y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1 = 0  (13.64)

But as β_{n} = C − α_{n} > 0, ξ_{n} = 0 and the equation above becomes

y_{n}(w^{T} x_{n} + b) − 1 = 0  (13.65)

• If α_{n} = C ≠ 0, then we have

y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1 = 0  (13.66)

But as β_{n} = C − α_{n} = 0, ξ_{n} ≥ 0 and the equation above becomes

y_{n}(w^{T} x_{n} + b) − 1 ≤ 0  (13.67)

In all three cases the expression y_{n}(w^{T} x_{n} + b) − 1 can be rewritten as

y_{n}(w^{T} x_{n} + b) − 1 = y_{n}(w^{T} x_{n} + b) − y_{n}^{2} = y_{n}[(w^{T} x_{n} + b) − y_{n}] = y_{n} E_{n} (13.68)

where E_{n} = (w^{T} x_{n} + b) − y_{n} is the error, or the difference between the actual and desired output We summarize all three cases above to get:

y_{n} E_{n} \{ \begin{matrix}≥ 0 & if α_{n} = 0 \\ = 0 & if 0 < α_{n} < C \\ ≤ 0 & if α_{n} = C \end{matrix}  (13.69)

These results derived from the KKT conditions in Eqs. (13.57) through (13.62) are an alternative form of the KKT conditions, which will be used in SMO algorithm to check if the KKT conditions are violated by any α_{n}.

The dual of the primal problem in Eq. (13.56) can be obtained by substituting the first set of equations (stationarity) into the primal Lagrangian:

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m}x_{m}^{T}x_{n}

+ C ∑_{n=1}^{N} ξ_{n} − ∑_{n=1}^{N} α_{n}ξ_{n} − ∑_{n=1}^{N} β_{n}ξ_{n}

= ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m}x_{m}^{T}x_{n}

subject to: 0 ≤ α_{n} ≤ C, ∑_{n=1}^{N} α_{n}y_{n} = 0  (13.70)

Note that due to the condition C = α_{n}+β_{n}, the Lagrangian multipliers ξ_{n} and β_{n} no longer appear in the Lagrangian L_{d} of the dual problem, which happens to be identical to that of the linearly separable problem discussed in previous section. Also, since α_{n} ≥ 0, β_{n} ≥ 0 and C = α_{n} + β_{n}, we have 0 ≤ α_{n} = C − β_{n} ≤ C. Having solved this QP problem for α_{1}, ⋯, α_{N}, we get the optimal decision plane in terms of the normal vector of the decision plane

w = ∑_{n=1}^{N} y_{n}α_{n}x_{n} = ∑_{n∈sv} y_{n}α_{n}x_{n}  (13.71)

based on Eq. (13.25), and the bais term b based on Eq. (13.27):

b = y_{n} − ∑_{i∈sv} y_{i}α_{i}x_{i}^{T} x_{n} = y_{n} − ∑_{i∈sv} y_{i}α_{i}K(x_{i}, x_{n}), (n ∈ sv)  (13.72)

where the inner product x_{k}^{T} x is replaced by the kernel function K(x_{k}, x) if it is nonlinear. Computationally, we take the average of such b values of all support vectors as the offset term.

Given w and b, any unlabeled point x can be classified into either of the two classes depending on whether it is on the positive or negative side of the decision plane:

w^{T} x + b = ∑_{i∈sv} y_{i}α_{i}x_{i}^{T} x + b = ∑_{i∈sv} y_{i}α_{i}K(x_{i}, x) + b \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix} (13.73)

where the inner product is replaced by a kernel function if it is nonlinear.

13.4 Sequential Minimal Optimization Algorithm

The sequential minimal optimization (SMO) is a more efficient algorithm for solving the SVM problem, compared with the generic QP algorithms such as the internal-point method. The SMO algorithm can be considered as a method of decomposition, by which an optimization problem of multiple variables is decomposed into a series of subproblems each optimizing an objective function of a small number of variables, typically only one, while all other variables are treated as constants that remain unchanged in the subproblem. For example, the coordinate descent algorithm is just such a decomposition method, which solves a problem in a multi-dimensional space by converting it into a sequence of subproblems each in a one-dimensional space.

When applying the decomposition method to the soft margin SVM problem, we could consider optimizing only one variable α_{i} at a time, while treating all remaining variables α_{1}, ⋯, α_{i−1}, α_{i+1}, ⋯, α_{N} as constants. However, due to the constraint ∑_{n=1}^{N} α_{n}y_{n} = 0, α_{i} is a linear combination of N − 1 constants and is therefore also a constant. We therefore need to consider two variables at a time, here assumed to be α_{i} and α_{j}, while treating the remaining N −2 variables as constants. The objective function of such a subproblem can be obtained by dropping all constant terms independent of the two selected variables α_{i} and α_{j} in the original objective function in Eq. (13.70):

maximize: L(α_{i}, α_{j}) = α_{i} + α_{j} − \frac{1}{2} (α_{i}^{2}x_{i}^{T} x_{i} + α_{j}^{2}x_{j}^{T} x_{j} + 2α_{i}α_{j}y_{i}y_{j}x_{i}^{T} x_{j})

−α_{i}y_{i} (∑_{n≠i} α_{n}y_{n}x_{n}^{T} ) x_{i} − α_{j}y_{j} (∑_{n≠j} α_{n}y_{n}x_{n}^{T} ) x_{j}

= α_{i} + α_{j} − \frac{1}{2} (α_{1}^{2}K_{ii} + α_{2}^{2}K_{jj} + 2α_{i}α_{j}y_{i}y_{j}K_{ij})

−α_{i}y_{i} ∑_{n≠i,j} α_{n}y_{n}K_{ni} − α_{j}y_{j} ∑_{n≠i,,j} α_{n}y_{n}K_{nj}

subject to: 0 ≤ α_{i}, α_{j} ≤ C, ∑_{n=1}^{N} α_{n}y_{n} = 0  (13.74)

where, to use the kernel method, all inner products x_{m}^{T}x_{n} are replaced by the corresponding kernel function K_{mn} = K(x_{m}, x_{n}) = z_{m}^{T}z_{n} with z_{n} = ϕ(x_{n}).

This maximization problem can be solved iteratively. Out of all previous values α_{n}^{old} (n = 1, ⋯, N), the two selected variables are updated to their correspoinding new values α_{i}^{new} and α_{j}^{new} that maximize L(α_{i}, α_{j}), subject to the two constraints.

We rewrite the second constraint as

α_{i}y_{i} + α_{j}y_{j} = − ∑_{n≠i,j} α_{n}y_{n}  (13.75)

and multiply both sides by y_{i} to get

y_{i}^{2}α_{i} + y_{i}y_{j}α_{j} = α_{i} + sα_{j} = (− ∑_{n≠i,j} α_{n}y_{n}) y_{i} = δ, i.e. α_{i} = δ − sα_{j}

(13.76)

where we have defined s = y_{i}y_{j} and δ = − ∑_{n≠i,j} α_{n}y_{n}y_{i}. As δ is independent of α_{i} and α_{j}, it remains the same before and after they are updated, and we have

α_{i}^{new} + sα_{j}^{new} = α_{i}^{old} + sα_{j}^{old} = δ  (13.77)

i.e.,

Δα_{i} = α_{i}^{new} − α_{i}^{old} = −s(α_{j}^{new} − α_{j}^{old}) = −sΔα_{j}  (13.78)

We now consider a closed-form solution for updating α_{i} and α_{j} in each iteration of this two-variable optimization problem. We first rewrite Eq. (13.25) as

∑_{n≠i,j} α_{n}y_{n}x_{n} = w − α_{i}y_{i}x_{i} − α_{j}y_{j}x_{j}  (13.79)

so that the two summations in L(α_{i}, α_{j}), now denoted by v_{k} (k = i, j), can be written as

v_{k} = ∑_{n≠i,j} α_{n}y_{n}K_{nk} = ∑_{n=1}^{N} α_{n}y_{n}K_{nk} − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}

= (∑_{n=1}^{N} α_{n}y_{n}K_{nk} + b) − b − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}
= u_{k} − b − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}  (13.80)

where

u_{k} = f(z_{k}) = ∑_{n=1}^{N} α_{n}y_{n}K_{nk} + b  (13.81)

is the output in Eq. (13.53) corresponding to input x_{k}. Now the objective function above can be written as:

L(α_{i}, α_{j}) = α_{i} + α_{j} − \frac{1}{2}(α_{i}^{2}K_{ii} + α_{j}^{2}K_{jj} + 2sα_{i}α_{j}K_{ij}) − α_{i}y_{i}v_{i} − α_{j}y_{j}v_{j} (13.82)

Substituting α_{i} = δ − sα_{j} (Eq. (13.78)) into L(α_{i}, α_{j}), we can rewrite it as a function of a single variable α_{j} alone:

L(α_{j}) = δ + (1 − s)α_{j} − \frac{1}{2} [(δ − sα_{j})^{2}K_{ii} + α_{j}^{2}K_{jj} + 2s(δ − sα_{j})α_{j}K_{ij}]
−(δ − sα_{j})y_{i}v_{i} − α_{j}y_{j}v_{j}
= \frac{1}{2}(2K_{ij} − K_{ii} − K_{jj})α_{j}^{2} + [1 − s + sδ(K_{ii} − K_{ij}) + y_{j}(v_{i} − v_{j})]α_{j} + Const. (13.83)

where s^{2} = (y_{i}y_{j})^{2} = 1, and scalar Const. represents all constant terms independent of α_{i} and α_{j} and can therefore be dropped. Now the objective function becomes a quadratic function of the single variable α_{j}. Consider the coefficients of this quadratic function:

• The coefficient of α_{j}^{2}:

\frac{1}{2}(2K_{ij} − K_{ii} − K_{jj}) = \frac{1}{2}η  (13.84)

where η is defined as

η = 2K_{ij} − K_{ii} − K_{jj} = 2z_{i}^{T} z_{j} − z_{i}^{T} z_{i} − z_{j}^{T} z_{j} = −(z_{i} − z_{j})^{T} (z_{i} − z_{j})
= −||z_{i} − z_{j}||^{2} ≤ 0  (13.85)

• The coefficient of α_{j}:

1 − s + sδ(K_{ii} − K_{ij}) + y_{j}(v_{i} − v_{j})
= 1 − s + (sα_{i} + α_{j})(K_{ii} − K_{ij})

+y_{j}[(u_{i} − b − α_{i}y_{i}K_{ii} − α_{j}y_{j}K_{ij}) − (u_{j} − b − α_{i}y_{i}K_{ij} − α_{j}y_{j}K_{jj})]

= 1 − s + sα_{1}(K_{ii} − K_{ij}) + α_{2}(K_{ii} − K_{ij})
+y_{j}(u_{i} − u_{j}) − α_{i}sK_{ii} − α_{j}K_{ij} + α_{i}sK_{ij} + α_{j}K_{jj}
= y_{j}^{2} − s − α_{j}(2K_{ij} − K_{ii} − 2K_{jj}) + y_{j}(u_{i} − u_{j})
= y_{j}(u_{i} − y_{j} − u_{j} + y_{j}) − α_{j}(2K_{ij} − K_{ii} − K_{jj})
= y_{j}(E_{i} − E_{j}) − α_{j}η  (13.86)

where we have defined

E_{k} = u_{k} − y_{k} = ∑_{n=1}^{N} α_{n}^{old}y_{n}K_{nk} + b − y_{k} (k = 1, 2)  (13.87)

as the difference between the desired output y_{k} and the actual output based on the previous values of the variables α_{n}^{old} (n = 1, ⋯, N).

Now the quadratic objective function can be rewritten as:

L(α_{j}) = \frac{1}{2}ηα_{j}^{2} + [y_{j}(E_{i} − E_{j}) − α_{j}^{old}η] α_{j}  (13.88)

Given the previous values α_{n}^{old} (n = 1, ⋯, N) in the coefficients, we will find α_{j}^{new} that maximizes L(α_{j}). Consider its first and second order derivatives:

\frac{d}{dα_{j}} L(α_{j}) = ηα_{j} + y_{j}(E_{i} − E_{j}) − α_{j}^{old}η
\frac{d^{2}}{dα_{j}^{2}} L(α_{j}) = η ≤ 0  (13.89)

As the second order derivative of L(α_{j}) is η ≤ 0, it has a maximum. Solving the equation d L(α_{j})/dα_{j} = 0, we get α_{j}^{new} that maximizes L(α_{j}) based on the old value α_{j}^{old}:

α_{j}^{new} = α_{j}^{old}+\frac{y_{j}(E_{j} − E_{i})}{η} = α_{j}^{old}+Δα_{j}, Δα_{j} = α_{j}^{new}−α_{j}^{old} = \frac{y_{j}(E_{j} − E_{i})}{η},

(13.90)

Note that this is an unconstrained solution, which needs to be modified if the constraints in Eq. (13.74) are violated. As shown in the figure below, α_{i} and α_{j} are inside the square of size C, due to the first constraint 0 ≤ α_{i}, α_{j} ≤ C; also, due to the second constraint α_{i} + sα_{j} = δ (Eq. (13.76)), they have to be on the diagonal line segment inside the square.

The four panels correspond to the four possible cases, depending on whether s = 1 (y_{i} = y_{j} = ±1, panels 1 and 2), or s = −1 (y_{i} = −y_{j} = ±1, panels 3 and 4), and whether δ < C (panels 1 and 3) or δ > C (panels 2 and 4), which can be further summarized below in terms of the lower bound L and upper bound H of α_{2}:

• If s = 1, then α_{i} + sα_{j} = α_{i} + α_{j} = δ, – if δ < C, then min(α_{j}) = 0, max(α_{j}) = δ (first panel)

– if δ > C, then min(α_{j}) = δ − C, max(α_{j}) = C (second panel)

Combining these two cases, we have

\{ \begin{matrix}L = max(0, δ − C) = max(0, α_{i} + α_{j} − C) \\ H = min(δ, C) = min(α_{i} + α_{j}, C) \end{matrix}  (13.91)

• If s = −1, then α_{i} + sα_{j} = α_{i} − α_{j} = δ, – if δ > 0, then min(α_{j}) = 0, max(α_{j}) = C − δ (third panel)

– if δ < 0, then min(α_{j}) = −δ, max(α_{j}) = C (fourth panel)

Combining these two cases, we have

\{ \begin{matrix}L = max(0, −δ) = max(0, α_{j} − α_{i}) \\ H = min(C − δ, C) = min(C + α_{j} − α_{i}, C) \end{matrix}  (13.92)

Now we apply the constraints in terms of L and H to the optimial solution α_{j}^{new} obtained previously to get α_{j} that is feasible as well as optmal:

α_{j}^{new} ⟸ \{ \begin{matrix}H & if α_{j}^{new} ≥ H \\ α_{j}^{new} & if L < α_{2}^{new} < H \\ L & if α_{j}^{new} ≤ L \end{matrix}  (13.93)

We can further find α_{i} based on Eq. (13.78):

α_{i}^{new} = α_{i}^{old} − sΔα_{j} = α_{i}^{old} − s(α_{j}^{new} − α_{j}^{old})  (13.94)

and update the weight vector based on Eq. (13.57):

w^{new} = ∑_{n≠i,j} y_{n}α_{n}^{old}x_{n} + y_{i}α_{i}^{new}x_{i} + y_{j}α_{j}^{new}x_{j}
= w^{old} − y_{i}α_{i}^{old}x_{i} − y_{j}α_{j}^{old}x_{j} + y_{i}α_{i}^{new}x_{i} + y_{j}α_{j}^{new}x_{j}
= w^{old} + y_{i}Δα_{i}x_{i} + y_{j}Δα_{j}x_{j} = w^{old} + Δw  (13.95)

where Δw = w^{new} − w^{old} = y_{i}Δα_{i}x_{i} + y_{j}Δα_{j}x_{j}. To update b, consider:

ΔE_{k} = E_{k}^{new} − E_{k}^{old} = u_{k}^{new} − u_{k}^{old}
= ∑_{n=1}^{N} α_{n}^{new}y_{n}K_{nk} + b^{new} − ∑_{n=1}^{N} α_{n}^{old}y_{n}K_{nk} − b^{old}
= y_{i}Δα_{i}K_{ik} + y_{j}Δα_{j}K_{jk} + b^{new} − b^{old} (k = 1, 2) (13.96)

Consider the following cases:

• If 0 < α_{k} < C for either k = i or k = j, then y_{k}E_{k} = 0 according to Eq. (13.69). We let E_{k}^{new} = 0 and solve the equation above to get:

b_{k}^{new} = b_{k}^{old} − (E_{k}^{old} + y_{i}Δα_{i}K_{ik} + y_{j}Δα_{j}K_{jk}) (k = 1 or 2) (13.97)

• If 0 < α_{k} < C for both k = i and k = j, then b_{i}^{new} = b_{j}^{new}.

• If α_{k} = 0 or α_{k} = C for both k = i and k = 2, then E_{k} ≠ 0, and b_{i}^{new} ≠ b_{j}^{new}, their average b^{new} = (b_{i}^{new} + b_{j}^{new})/2 can be used.

The computation above for maximizing L(α_{i}, α_{j}) is carried out iteratively each time when a pair of variables α_{i} and α_{j} is selected to be updated. Specifically, we select a variable α_{i} that violates any of the KKT conditions given in Eq. (13.69) in the outer loop of the SMO algorithm, and then pick a second variable α_{j} in the inner loop of the algorithm, both to be optimized. The selection of these variables can be either random (e.g., by following their order), or guided by some heuristics, such as always choosing the variable with maximum step size Δα_{n} = α_{n}^{new} −α_{n}^{old}. This iterative process is repeated until convergence when the KKT conditions are satisfied by all α_{1}, ⋯, α_{N} variables, i.e., no more variables need to be updated. All data points corresponding to α_{n} ≠ 0 are the support vectors, based on which we can find the offset b by Eq. (13.72) and classify any unlabeled point x by Eq. (13.73).

The Matlab code for the essential part of the SMO algorithm is listed below, based mostly on a simplified SMO algorithm with some modifications.

In particular, the if-statement

if (alpha(i)<C & yE<-tol) | (alpha(i)>0 & yE>tol)

checks the three KKT conditions in Eq. (13.69), where yE = y_{n}E_{n} and tol is a small constant. The first half checks if the first two of the three conditions are violated, while the second half checks if the second two of the three conditions are violated.

    [X y]=getData;                       % get training data
    [m n]=size(X);                       % size of data
    alpha=zeros(1,n);                    % alpha variables
    bias=0;                              % initial bias
    it=0;                                % iteration index
    while (it<maxit)                     % number of iterations less than maximum
        it=it+1;
        changed_alphas=0;                % number of changed alphas
        N=length(y);                     % number of support vectors
        for i=1:N                        % for each alpha_i
            Ei=sum(alpha.*y.*K(X,X(:,i)))+bias-y(i);
            yE=Ei*y(i);
            if (alpha(i)<C & yE<-tol) | (alpha(i)>0 & yE>tol)   % KKT violation
                for j=[1:i-1,i+1:N]        % for each alpha_j not equal alpha_i
                    Ej=sum(alpha.*y.*K(X,X(:,j)))+bias-y(j);
                    ai=alpha(i);         % alpha_i old
                    aj=alpha(j);         % alpha_j old
                    if y(i)==y(j)        % s=y_i y_j=1

    L=max(0,alpha(i)+alpha(j)-C);
    H=min(C,alpha(i)+alpha(j));
else                 % s=y_i y_j=-1
    L=max(0,alpha(j)-alpha(i));
    H=min(C,C+alpha(j)-alpha(i));
end
if L==H              % skip when L==H
    continue
end
eta=2*K(X(:,j),X(:,i))-K(X(:,i),X(:,i))-K(X(:,j),X(:,j));
alpha(j)=alpha(j)+y(j)*(Ej-Ei)/eta;   % update alpha_j
if alpha(j) > H
    alpha(j) = H;
elseif alpha(j) < L
    alpha(j) = L;
end
if norm(alpha(j)-aj) < tol       % skip if no change
    continue
end
alpha(i)=alpha(i)-y(i)*y(j)*(alpha(j)-aj);   % find alpha_i
bi = bias - Ei - y(i)*(alpha(i)-ai)*K(X(:,i),X(:,i))...
    -y(j)*(alpha(j)-aj)*K(X(:,j),X(:,i));
bj = bias - Ej - y(i)*(alpha(i)-ai)*K(X(:,i),X(:,j))...
    -y(j)*(alpha(j)-aj)*K(X(:,j),X(:,j));
if 0<alpha(i) & alpha(i)<C
    bias=bi;
elseif 0<alpha(j) & alpha(j)<C
    bias=bj;
else
    bias=(bi+bj)/2;
end
changed_alphas=changed_alphas+1;  % one more alpha changed
end
end
end
if changed_alphas==0             % no more changed alpha, quit
break
end
I=find(alpha~=0);                % indecies of non-zero alphas
alpha=alpha(I);                  % find non-zero alphas
Xsv=X(:,I);                      % find support vectors
ysv=y(I);                        % their corresponding indecies
end                                  % end of iteration
nsv=length(ysv);                     % number of support vectors

where K(X,x) is a function that takes an m × n matrix X = [x_{1}, ⋯, x_{n}] and an n-D vector x and produces the kernel functions K(x_{i}, x), (i = 1, ⋯, n) as output.

function ker=K(X,x)
    global kfunction
    global gm                               % parameter for Gaussian kernel
    [m n]=size(X);
    ker=zeros(1,n);
    if kfunction=='l'
        for i=1:n
            ker(i)=X(:,i).'*x;               % linear kernel
        end
    elseif kfunction=='g'
        for i=1:n
            ker(i)=exp(-gm*norm(X(:,i)-x)); % Gaussian kernel
        end
    elseif kfunction=='p'
        for i=1:n
            ker(i)=(X(:,i).'*x).^3;          % polynomial kernel
        end
    end
end

Having found all non-zero α_{n} ≠ 0 and the corresponding support vectors, we further find the offset or bias term:

     bias=0;
     for i=1:nsv
        bias=bias+(ysv(i)-sum(ysv.*alpha.*K(Xsv,Xsv(:,i))));
    end
    bias=bias/nsv;

Any unlabeled data point x is classified into class C_{+} or C_{−}, depending on whether the following expression is greater or smaller than zero:

    y=sum(alpha.*ysv.*K(Xsv,x))+bias;

Example 1: The same training set used previously to test the SVM with a hard margin is used again to test the SVM with a soft margin. Two results are shown below, corresponding to two different values C = 1.2 and C = 0.05 for the weight of the error term ∑_{n=1}^{N} ξ_{n} in the objective function. We see that when C = 1.2 is large, the number of support vectors (the four solid dots) is small due to the small errors ξ_{n} allowed, and the decision boundary determined by the these support vectors independent of the rest of the data set (circles) is mostly dictated by the local points close to the boundary, not necessarily a good reflection of the global distribution of the data points. This result is similar to the previous one generated by the hard-margin SVM.

On the other hand, when C = 0.1 is small, a greater number of data points become support vectors due to the larger (the 13 solid dots) error ξ_{n} allowed, and the resulting linear decision line determined by these support vectors reflects global distribution of the data points, and it better separates the two classes in terms of their mean positions.

Example 2: The classification results for the XOR data set are shown below with C = 1.2 and C = 0.05. As the two classes in the XOR data set are not linearly separable, a Gaussian or radial basis function (RBF) kernel is used to map the data from 2-D space to an infinite dimensional space, which is partitioned into two regions for the two classes. When C = 1.2 is large, the decision boundary is dictated by a relatively small number of support vectors (solid dots, with small error ξ_{n}) and all data points are classified correctly, but it is more prone to the overfitting problem, if some of the small number of support vectors are outliers due to noise.

One the other hand, when C = 0.05 is small, the decision boundary is determined by a greater number of support vectors, better reflecting the global distribution of the data points. However, as the local points close to the boundary are not relatively deemphasized, some miss-classification occurs, some nine red dots are classified into the blue region incorrectly.

Example 3:

Example 4:

13.5 Multi-Class Classification

The SVM method is inherently a binary classifier, but it can be adapted to classification problems of more than two classes. In the following, we consider a general K-class classifier based on a training set X = [x_{1}, ⋯, x_{N}], of which each sample x_{n} is labeled by y_{n} ∈ {1, ⋯, K} to indicate x_{n} ∈ C_{y_{n}}.

We first consider two straight forward and imperical methods for multiclass classification based directly on binary SVM.

• One-Vs-One (1V1)

Any unlabeled x is classified to a class which receives the maximum votes out of the K(K − 1)/2 binary classifications between every pair of the K classes.

• One-Vs-Rest (1VR)

This method converts a K-class problem (K > 2) into K binary problems. First, we regroup the K classes C_{1}, ⋯, C_{K} into two classes C_{+} = C_{k} and C_{−} = {C_{l}|l = 1, ⋯, K, l ≠ k} and find the corresponding decision function f_{k}(x) representing quantitatively to what extent a given x belongs to class C_{+} = C_{k} (if f_{k}(x) > 0), instead of C_{−} containing all remaining K − 1 classes (if f_{k}(x) < 0). After this process has been carried out for all k = 1, ⋯, K, the x can be classified as below:

if f_{l}(x) = max{f_{1}(x), ⋯, f_{K}(x)}, then x ∈ C_{l}  (13.98)

We further consider another method for multiclass classifiction by directly generalizing the binary SVM so that the decision margin between any two of the K classes is maximized. First, we define a linear function for each of the K classes:

f_{k}(x) = w_{k}^{T} x + b_{k} (k = 1, ⋯, K)  (13.99)

Once the parameters w_{k} and b_{k} are determined during the training set, any unlabeled point x can be classified as below:

if f_{l}(x) = max_{k}{f_{k}(x) (k = 1, ⋯, K)}, then x ∈ C_{l}  (13.100)

The figure below illustrates the classification of K = 3 classes in d = 2 dimensional feature space. Here each straight lines H_{k} (plane or hyperplane if d ≥ 3) is determined by equation f_{k}(x) = w_{k}^{T} x + b_{k} = 0, (k = 1, ⋯, K), with normal direction in w_{k} and distance d(H_{k}, 0) = |b_{k}|/||w_{k}|| to the origin (same as in Eq. (13.4)).

According to the classification rule in Eq. (13.100), the 2-D feature space is partitioned into three regions each for one of the K = 3 classes by the decision boundaries B_{12}, B_{23}, B_{31}, of which boundary B_{ij} between classes C_{i} and C_{j} is composed of all points x ∈ B_{ij} satisfying f_{i}(x) = f_{j}(x). Same as in binary SVM, we further define for each of the K classes two additional straight lines (planes or hyperplanes if d ≥ 3) parallel to H_{k}, denoted by H_{k±} and represented by equations f_{k}(x) = w_{k}^{T} x + b_{k} ± c_{k}. Their distances to H_{k} can be found as (same as in Eq. (13.9)):

d(H_{k±}, H_{k}) = \frac{|c_{k}|}{||w_{k}||}  (13.101)

Also as shown in the figure, the intersections of these parallel lines for classes C_{1} and C_{2} futher determine two straight lines parallel to the decision boundary B_{12}, defining the decision margin between the support vectors of C_{1} and C_{2}. It can be seen that this decision margin is monotonically related to both |c_{1}|/||w_{1}|| and |c_{2}|/||w_{2}||, which can be maximized by minimizing both ||w_{1}|| and ||w_{2}||.

Now the multiclass classification problem can be formulated as to find the parameters w_{k} and b_{k} for all k = 1, ⋯, K so that ∑_{k=1}^{K} ||w_{k}||^{2} is minimized and thereby the decision margins for all boundaries B_{ij} between C_{i} and C_{j} are maximized, under the constraint that every training sample x_{n} labeled by y_{n} is correctly classified with a decision margin of 1 (the distance between the support vectors on both sides of B_{ij} is 2):

minimize: \frac{1}{2} ∑_{k=1}^{K} ||w_{k}||^{2}
subject to: w_{y_{n}}^{T}x_{n} + b_{y_{n}} ≥ w_{k}^{T} x_{n} + b_{k} + 2, (k = 1, ⋯, K, k ≠ y_{n}) (13.102)

Similar to Eq. (13.54) in soft margin SVM, the contraints of the optimization problem above can be relaxed by including in each contraint an extra error term ξ_{nk} ≥ 0, which is to be minimized. Now the problem can be reformulated as

minimize: \frac{1}{2} ∑_{k=1}^{K} ||w_{k}||^{2} + C ∑_{k=1}^{K} ∑_{n=1}^{N} ξ_{nk}
subject to: w_{y_{n}}^{T}x_{n} + b_{y_{n}} ≥ w_{k}^{T} x_{n} + b_{k} + 2 − ξ_{nk}
ξ_{nk} ≥ 0
(n = 1, ⋯, N, k = 1, ⋯, K, k ≠ y_{n})  (13.103)

The Lagrangian function of this constrained optimization problem is

L(w, b, ξ, α, β) = \frac{1}{2} ∑_{k=1}^{K} ||w_{k}||^{2} + C ∑_{k} ∑_{n} ξ_{nk}
− ∑_{k} ∑_{n} α_{nk} [(w_{y_{n}} − w_{k})^{T} x_{n} + b_{y_{n}} − b_{k} − 2 + ξ_{nk}] − ∑_{k} ∑_{n} β_{nk}ξ_{nk} (13.104)

where

α_{ny_{n}} = β_{ny_{n}} = 0, ξ_{ny_{n}} = 2;
α_{nk} ≥ 0, β_{nk} ≥ 0, ξ_{nk} ≥ 0
(n = 1, ⋯, N, k = 1, ⋯, K, k ≠ y_{n})  (13.105)

\frac{∂}{∂w_{k}} L(w, b, ξ, α, β) = ∑_{k} w_{k} + ∑_{n} α_{nk}x_{n} = 0  (13.106)

\frac{∂}{∂b_{k}} L(w, b, ξ, α, β) = ∑_{k} ∑_{n} α_{nk} = 0  (13.107)

\frac{∂}{∂ξ_{nk}} L(w, b, ξ, α, β) = C − α_{nk} − β_{nk} = 0  (13.108)

f(x) = W^{T} x  (13.109)

where

W = [w_{1}, ⋯, w_{K}]  (13.110)

An unlabeled sample x is classified to class C_{i} if f_{i}(x) = max(f_{1}(x), ⋯, f_{K}(x)).

The training of the classifier is to find all weight vectors in W based on the training dataset. To do so, we first define a cost function

////

13.6 Kernelized Bayes classifier

The naive Bayes classification is based on a quadratic decision function and is therefore unable to classify data that are not quadratically separable. However, as shown below, the Bayes method can be reformulated so that all data points appear in the form of an inner product, and the kernel method can be used to map the original space into a higher dimensional space in which all groups can be separated even though they are not quadratically separable in the original space.

Consider a binary Bayes classifier by which any sample x is classified into either of the two classes C_{+} and C_{−} depending on whether x is on the positive or negative side of the quadratic decision surface (Eq. (12.22)):

f(x) = x^{T} Wx + w^{T} x + w \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix}  (13.111)

As show in Eq. (12.21), here

W = −\frac{1}{2}(Σ_{+}^{−1} − Σ_{−}^{−1})
w = Σ_{+}^{−1}m_{+} − Σ_{−}^{−1}m_{−}
w = −\frac{1}{2}(m_{+}^{T}Σ_{+}^{−1}m_{+} − m_{−}^{T}Σ_{−}^{−1}m_{−}) − \frac{1}{2} log \frac{|Σ_{+}|}{|Σ_{−}|} + log \frac{P (C_{+})}{P (C_{−})}(13.112)

where Σ_{+} and Σ_{−} are respectively the covariance matrices of the samples in C_{+} and C_{−},

m_{+} = \frac{1}{N_{+}} ∑_{x∈C_{+}} x, m_{−} = \frac{1}{N_{−}} ∑_{x∈C_{−}} x  (13.113)

are their mean vectors, and P_{+} = N_{+}/N and P_{−} = N_{−}/N are their prior probabilities. Specially if Σ_{+} = Σ_{−} = Σ and therefore W = 0, the quadratic decision surface becomes a linear decision plane discribed by

f(x) = w^{T} x + w = [Σ^{−1}(m_{+} − m_{−})]^{T} x + w = 0  (13.114)

in the same form as the decision equation for the support vector machine. An unlabeled sample x is classified into either of the two classes depending on on which side of a threshold −w its projection onto the normal direction w of the decision plane lies.

As matrices W, Σ_{+}^{−1}, and Σ_{−}^{−1} are all symmetric, they can be written in the following eigendecomposition form:

W = VΛV^{T} = UU^{T} , Σ_{+}^{−1} = V_{+}Λ_{+}V_{+}^{T} = U_{+}U_{+}^{T}, Σ_{−}^{−1} = V_{−}Λ_{−}V_{−}^{T} = U_{−}U_{−}^{T}

(13.115)

where U = VΛ^{1/2}, U_{+} = V_{+}Λ_{+}^{1/2} and U_{−} = V_{−}Λ_{−}^{1/2}. We can write vector w as:

w = Σ_{+}^{−1}m_{+} − Σ_{−}^{−1}m_{−} = U_{+}U_{+}^{T} \frac{1}{N_{+}} ∑_{x_{+}∈C_{+}} x_{+} − U_{−}U_{−}^{T} \frac{1}{N_{−}} ∑_{x_{−}∈C_{−}} x_{−}

(13.116)

Any unlabeled sample x can now be classified into either of the two classes based on its decision function:

f(x) = x^{T} Wx + w^{T} x + w
= x^{T} UU^{T} x + (\frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}U_{+}U_{+}^{T}) x − (\frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}U_{−}U_{−}^{T}) x + w

= z^{T} z + \frac{1}{N_{+}} ∑_{z_{++}} (z_{++}^{T}z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} (z_{−−}^{T}z_{−}) + w  (13.117)

where

\{ \begin{matrix}z_{++} = U_{+}^{T}x_{+} (x_{+} ∈ C_{+}) \\ z_{−−} = U_{−}^{T}x_{−} (x_{−} ∈ C_{−}) \end{matrix} , \{ \begin{matrix}z = U^{T} x \\ z_{+} = U_{+}^{T}x \\ z_{−} = U_{−}^{T}x \end{matrix}  (13.118)

As all data points appear in the form of an inner product, the kernel method can be used by replacing all inner products in the decision function by the corresponding kernel functions:

f(x) = K(z, z) + \frac{1}{N_{+}} ∑_{z_{++}} K(z_{++}, z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} K(z_{−−}, z_{−}) + b = p(x) + b

(13.119)

This is the decision function in the new higher dimensional space, where p(x) is defined as a function composed of all terms in f(x) except the last offset term b, which is to replace w in the original space. To find this b, we first map all training samples into a 1-D space x_{n} = p(x_{n}), (n = 1, ⋯, N) and sort them together with their corresponding labelings y_{1}, ⋯, y_{N}, and then search through all N −1 possible ways to partition them into two groups indexed respectively by 1, ⋯, k and k + 1, ⋯, N to find the optimal k corresponding to the maximum labeling consistency measured by

| ∑_{n=1}^{k} y_{n}| + | ∑_{n=k+1}^{N} y_{n}|, (k = 1, ⋯, N − 1)  (13.120)

The middle point (x_{k} + x_{k+1})/2 between x_{k} and x_{k+1} is used as the optimal threshold to separate the training samples into two classes in the 1-D space, i.e., the offset is b = −(x_{k} + x_{k+1})/2. Now the unlabeled point x can be classified into either of the two classes C_{+} and C_{−}:

p(x) + b \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix}  (13.121)

In general, when all data points are kernel mapped to a higher dimensional space, the two classes can be more easily separated, even by a hyperplane based on the linear part of the decision function without the second order term. This allows the assumption that the two classes have the same covariance matrix so that W = −(Σ_{+} − Σ_{−})/2 = 0 and the second order term is dropped. This is the justification for the following two special cases:

• If we approximate both Σ_{+} and Σ_{−} by their average Σ = (Σ_{+} +Σ_{−})/2, then W = 0 and the decision function of any x becomes

f(x) = w^{T} x + w = [Σ^{−1}(m_{+} − m_{−})]^{T} x + w
= [UU^{T} (\frac{1}{N_{+}} ∑_{x_{+}} x_{+} − \frac{1}{N_{−}} ∑_{x_{−}} x_{−})]^{T} x + w

= \frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}UU^{T} x − \frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}UU^{T} x + w
= \frac{1}{N_{+}} ∑_{z_{+}} z_{+}^{T}z − \frac{1}{N_{−}} ∑_{z_{−}} z_{−}^{T}z + w  (13.122)

where UU^{T} = Σ^{−1}, z_{+} = U^{T} x_{+}, z_{−} = U^{T} x_{−}, and z = U^{T} x. This decision function can be converted to the following if the kernel method is used:

f(x) = \frac{1}{N_{+}} ∑_{z_{+}} K(z_{+}, z) − \frac{1}{N_{−}} ∑_{z_{−}} K(z_{−}, z) + b  (13.123)

• More specially, if Σ_{+} = Σ_{−} = I, then the decision function above becomes

f(x) = w^{T} x + w = (m_{+} − m_{−})^{T} x + w
= \frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}x − \frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}x + w  (13.124)

which can be converted to the following if the kernel method is used:

f(x) = \frac{1}{N_{+}} ∑_{x_{+}} K(x_{+}, x) − \frac{1}{N_{−}} ∑_{x_{−}} K(x_{−}, x) + b  (13.125)

Note that if the kernel method is used to replace an inner product by a kernel function K(x, x′) = ϕ(x)^{T} ϕ(x′), we need to map all data points to ϕ(x_{n}), (n = 1, ⋯, N) in the higher dimensional space, instead of only mapping their means ϕ(m_{+}) and ϕ(m_{−}), because the mapping of a sum is not equal to the sum of the mapped points if the kernel is not linear:

ϕ(m_{+}−m_{−}) ≠ ϕ(m_{+})−ϕ(m_{−}), ϕ(m_{±}) = ϕ (\frac{1}{n_{±}} ∑_{x∈C_{±}} x) ≠ \frac{1}{n_{±}} ∑_{x∈C_{±}} ϕ(x)

(13.126)

The three cases above are summarized below:

• Type I, Eq. (13.125)

f(x) = \frac{1}{N_{+}} ∑_{x_{+}} K(x_{+}, x) − \frac{1}{N_{−}} ∑_{x_{−}} K(x_{−}, x) + b  (13.127)

• Type II, Eq. (13.122)

f(x) = \frac{1}{N_{+}} ∑_{z_{+}} K(z_{+}, z) − \frac{1}{N_{−}} ∑_{z_{−}} K(z_{−}, z) + b  (13.128)

• Type III, Eq. (13.119)

f(x) = z^{T} z + \frac{1}{N_{+}} ∑_{z_{++}} K(z_{++}, z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} K(z_{−−}, z_{−}) + b (13.129)

The Matlab code for the essential part of the algorithm is listed below. Given X and y for the data array composed of N training vectors X = [x_{1}, ⋯, x_{N}] and their corresponding labelings y = [y_{1}, ⋯, y_{N}], the code carries out the training and then classifies any unlabeled data point into either of the two classes. Parameter type selects any one of the three different versions of the algorithm, and the function K(X,x) returns a 1-D array containing all N kernel functions K(x_{i}, x), (i = 1, ⋯, N) of the column vectors in X and vector x.

    X=getData;
    [m n]=size(X);
    X0=X(:,find(y>0));   n0=size(X0,2);     % separate C+ and C-
    X1=X(:,find(y<0));   n1=size(X1,2);
    m0=mean(X0,2);    C0=cov(X0');          % calculate mean and covariance
    m1=mean(X1,2);    C1=cov(X1');
    if type==1
        for i=1:n
            x(i)=sum(K(X0,X(:,i)))/n0-sum(K(X1,X(:,i)))/n1;
        end
    elseif type==2
        C=inv(C0+C1);   [V D]=eig(C);   U=(V*D^(1/2))';
        Z=U*X;  Z0=U*X0;   Z1=U*X1;
        for i=1:n
            x(i)=sum(K(Z0,Z(:,i)))/n0-sum(K(Z1,Z(:,i)))/n1;
        end

elseif type==3
C0=inv(C0);   C1=inv(C1);   W=-(C0-C1)/2;
[V D]=eig(W);     U=(V*D^(1/2)).';    Z=U*X;
[V0 D0]=eig(C0);  U0=(V0*D0^(1/2))';  Z0=U0*X;  Z00=U0*X0;
[V1 D1]=eig(C1);  U1=(V1*D1^(1/2))';  Z1=U1*X;  Z11=U1*X1;
for i=1:n
    x(i)=K(Z(:,i),Z(:,i))+sum(K(Z00,Z0(:,i)))/n0-sum(K(Z11,Z1(:,i)))/n1;
end
end
[x I]=sort(x);   y=y(I);    % sort 1-D data together with their labelings
smax=0;
for i=1:n-1                 % find optimal threshold value b
s=abs(sum(y(1:i)))+abs(sum(y(i+1:n)));
if s>smax
    smax=s;  b=-(x(i)+x(i+1))/2;
end
end

Note that W = −(Σ_{0}^{−1} − Σ_{1}^{−1})/2 may be either positive or negative definite, and its eigenvalue matrix D may contain negative values and D^{1/2} may contain complex values. Given any unlabeled data point x, the code below is carried out

if type==1
y(i)=sum(K(X0,x))/n0-sum(K(X1,x))/n1+b;
elseif type==2
Z=U*x;
y(i)=sum(K(Z0,z))/n0-sum(K(Z1,z))/n1+b;
elseif type==3
z=U*x;    z0=U0*x;   z1=U1*x;
y(i)=K(z,z)+sum(K(Z00,z0))/n0-sum(K(Z11,z1))/n1+b;
end

to classify x into either C_{+} if y > 0, or C_{−} if y < 0.

In comparison with the SVM method, which requires solving a QP problem by certain iterative algorithm (either the interior point method or the SMO method), the kernel based Bayes method is closed-form and therefore extremely efficient computationally. Moreover, as shown in the examples below, this method is also highly effective as its classification results are comparable and offen more accurate than those of the SVM method.

We now show a few examples to test all three types of the kernel based Bayes method based on a set of simulated 2-D data. Both linear kernel and RBF kernel K(x, y) = exp(−γ||x − y||). The value of the parameter γ used in the examples is 5, but it can be fine tuned in a wide range (e.g., 1 < γ < 9 to make proper trade-off between accuracy and avoiding overfitting. The performances of these method are also compared with the SVM method implemented by the Matlab function:

fitcsvm(X',y,'Standardize',true,'KernelFunction','linear','KernelScale','auto'))

Example 1: Based on some 2-D training datasets, four different binary classifiers are tested. The correct rates of each of the four methods are listed below, together with the corresponding partitionings of the 2-D space shown in the figures.

Kernel

Matlab SVM Kernel Bayes I Kernel Bayes II

Kernel Bayes III

Test 1 linear  93.0%  88.0%  94.0%  94.0%
Test 2 linear  73.0%  80.0%  79.5%  96.5%
Test 3 RBF  98.5%  100%  100%  100%

(13.130)

• Test 1: Two sets of data are generated based on the following mean vectors and covarience matrices:

m_{+} = [ \begin{matrix}−1 \\ 0 \end{matrix} ] , m_{−} = [ \begin{matrix}+1 \\ 0 \end{matrix} ] , Σ_{+} = Σ_{−} = 3 × [ \begin{matrix}1 & 0.5 \\ 0.5 & 0.5 \end{matrix} ]

(13.131)

The linear kernel is used for all methods. The kernel Bayes methods II and III achieve the best result (94%), slightly better than that of the standard SVM (93%). But the kernel Bayes method I performs poorly (84.7%), as it is based only on the means of the two classes without taking into consideration their covariances representing the distribution of the data points.

• Test 2: Two sets of data not linearly separable are classified by the four methods all based on the linear kernel. The kernel Bayes III achieves the best result (96.5%), due to its quadratic term in the decision function, the kernel Bayes I and II perform much more poorly, but still slightly better than the SVM method.

• Test 3: The same dataset as above is classified by the methods but now based on the RBF kernel. While the SVM performs performs reasonably well (98.5%), all three versions of the kernel Bayes method (with γ = 5) achieve the perfect result with all points of the two classes completely separate. However, the partitioning of the space by the kernel Bayes III is highly fragmented, showing a sign of overfitting.

Example 2: The three datasets are generated using the Matlab code at

https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html.

A parameter value γ = 5 is used for the three versions of the kernel Bayes method.

Kernel

Matlab SVM Kernel Bayes I Kernel Bayes II

Kernel Bayes III

Test 1 Linear  58.75%  60.0%  60.25%  97.0%
Test 2 RBF  97.75%  98.50%  98.50%  99.50%
Test 3 RBF  98.0%  100%  100%  100%
Test 4 RBF  96.0%  98.5%  95.5%  97.0%

(13.132)

• Test 1: Exclusive OR dataset, linear kernel

As the data are not linearly separable, all methods performed poorly except kernelized Bayes Type III with a second order term in the decison function.

• Test 2: Exclusive OR dataset, RBF kernel

All four methods performed very well, but all three variations of the kernelized Bayes method achieved higher correct rates than the SVM.

• Test 3: Cocentric ring dataset, RBF kernel

• Test 4: Multi-cluster dataset, RBF kernel

Example 3: The classification results of Fisher’s iris data by the SVM method (Matlab function fitcsvm) and the kernel Bayes methods are shown below. This is an example used to illustrate the SVM method in the documentation of fitcsvm

(https://www.mathworks.com/help/stats/fitcsvm.html).

In this example only two (3rd and 4th) of the four features are used, with half of the samples used for training while the other half for testing. Note that the second class (setosa in green) and third class (versicolor in blue) not linearly separable can be better separated by the kernel Bayes method.

13.7 Hierarchical (Tree) Classifiers

Both supervised classification and unsupervised clustering can be carried out in a hierarchical fashion to classify the input patterns or group them into clusters, very much like the hierarchy of biological classifications with different taxonomic ranks (domain, kingdom, phylum, class, order, family, genus, and species).

• Unsupervised clustering

The hierarchical clustering can be obtained in either a top-down or bottom up manner.

– Top-down method:

All patterns in the data set are initially treated as a single cluster as the root of the tree, which is then subdivided (split) into a set of two or more smaller clusters, each represented as a node in the tree structure. This process is carried out recursively until eventually each cluster contains only one pattern, represented as a leaf node of the tree.

– Bottom-up method:

every pattern in the data set is initially treated as a cluster as a leaf node of the tree, which will then be merged to form larger clusters. Again, this process is carried out recursively until eventually all patterns are merged into a single cluster at the root of the tree.

In either the top-down or the bottom-up method, the specific method for the splitting or merging at each tree node is based on certain similarity measurement such as the distance between two clusters. The resulting tree structure obtained by either method can then be truncated at any level between the root and the leaf nodes to obtain a set of clusters, depending on the desired number and sizes of these clusters.

• Supervised classification

If labeled training data are available, both the top-down and the bottomup clustering methods can also be used in the training stage of the supervised classification methods, with the only difference that now the splitting or merging is applied to labeled classes instead of individual patterns, and each leaf node represents one of the classes, rather than a single pattern. After the tree structure is obtained, the training is complete and any unlabeled pattern can be classified at the tree root and then subsequently the tree nodes at lower levels until it is classified into one of the leaf nodes of the tree, corresponding to a specific class.

This hierarchical classification method is especially useful when the number of classes and the number D of feature are both large. In this case it may be very difficult to select a subset of d < D features good for separating all classes for a single-level classifier, by which all classes need to be classified at the same time, requiring, most likely, all N features. However, for a tree classifier, since each node is a two-class classifier, it is possible to select a small number of d ≪ D features that are most relevant and suitable to represent the two subsets of classes.

In the following we consider both the bottom-up and top-down methods for hierarchical clustering/classification.

Bottom-Up method The bottom-up hierarchical classifier is trained based on K classes C_{1}, ⋯, C_{K},

each containing n_{k} (k = 1, ⋯, K) labeled patterns x ∈ C_{k}.

1. Compute the K(K − 1)/2 pairwise Bhattacharyya distances between every two classes C_{i} and C_{j}:

d_{B}(C_{i}, C_{j}) = \frac{1}{4}(m_{i} − m_{j})^{T} [\frac{Σ_{i} + Σ_{j}}{2} ]^{−1} (m_{i} − m_{j}) + log [\frac{|\frac{Σ_{i}+Σ_{j}}{2} |}{(|Σ_{i}| |Σ_{j}|)^{1/2}} ]

(13.133)

2. Merge the two classes corresponding to the smallest d_{B} to form a new class C_{i} ∪ C_{j} = C_{k}, compute its mean and covariance:

m_{k} = \frac{1}{n_{i} + n_{j}} [n_{i}m_{i} + n_{j}m_{j}]  (13.134)

and

Σ_{k} = \frac{1}{n_{i} + n_{j}} [n_{i}(Σ_{i}+(m_{i}−m_{k})(m_{i}−m_{k})^{T})+n_{j}(Σ_{j}+(m_{j}−m_{k})(m_{j}−m_{k})^{T})]

(13.135)

Delete the old classes C_{i} and C_{j}. Now there are K − 1 classes left.

3. Compute the distance between the new class C_{k} and all K − 2 remaining classes.

4. Repeat the previous steps until eventually all classes are merged into a single group containing all K classes, the binary tree structure is thus obtained.

Top-Down method

Generate a binary tree by recursively partitioning all classes into two subgroups with the maximum Bhattacharyya distance

1. Compute the between-class scatter matrix S_{B} of the K classes, find its maximum eigenvalue λ_{i} and the corresponding eigenvectors v_{i};

2. Project all data points onto v_{1}:

y_{n} = x_{n}^{T} v (n = 1, ⋯, N)  (13.136)

3. Sort all data points {y_{1}, ⋯, y_{N}} along this 1-D space and partition them into two subgroups with maximum Bhattacharyya (between-group) distance.

4. Carry out the steps above recursively to each of the two subgroups, until eventually every subgroup contains only one classes

Once the hierarchical structure is constructed by either the bottom-up or topdown method, we need to build a binary classifier at each node of structure, by which any given pattern is classified into either the left group G_{l} or right group G_{r}:

• According to the specific classification method used, find the discriminant functions D_{l}(x) and D_{r}(x) for the two subgroups based on the training data.

• Select the best d < D features most suitable for separating the two groups G_{l} and G_{r}, based on any of the feature selection methods such as those listed below:

– Choosing d features directly from the D original ones using between-class distance (Bhattacharrya distance) as the criterion,

– Carry out KLT based on the between-class scatter matrix S_{B} and use the first d principal components for the binary classification. As here only two groups of classes need to be distinguished, the number of features M can be expected to be small.

• Any unlabeled pattern x enters the classifier at the root of the tree and is classified to either the left or the right sub-group of the node according to the discriminant function

x ∈ \{ \begin{matrix}G_{l} & if D_{l}(x) > D_{r}(x) \\ G_{r} & if D_{r}(x) < D_{l}(x) \end{matrix}  (13.137)

This process is carried out recursively at each of the subsequent nodes until eventually x reaches one of the leaf nodes corresponding to a single class, to which the sample x is therefore classified.

Example

The hierarchical clustering method is applied to a dataset composed of seven normally distributed clusters each containing 25 sample vectors in an N = 4 dimensional space. The PCA method is used to project the data in 4-D space into a 2-D space spanned by the first two principal components, as shown below:

The clustering result is shown below. Each column in the display represents the four components of a 4-D vector, color coded by a spectrum from red (low values) through green (middle) to blue (high values).

More examples of clustering analysis applied to gene data analysis in bioinfor-

matics can be found here: http://fourier.eng.hmc.edu/bioinformatics/intro/index.html
In particular, an example of this method is available here: http://fourier.eng.hmc.edu/bioinformatics/intro/node12.html.

14 Clustering Analysis

All classification algorithms discussed above are supervised in nature based on the assumped availability of some training data in which each of the data patterns x_{n} ∈ X = [x_{1}, ⋯, x_{N}}] is labeled by y_{n} ∈ y = [y_{1}, ⋯, y_{n}]^{T}, i.e., the class they each belong is known. However, when such training set is not available, there is still the need to explore some potential structure of the data in the form of a set of unknown number K of clusters {C_{1}, ⋯, C_{K}}, each composed of a set of similar data points close to each other in the feature space. This can be accomplished by the unsupervised method of clustering analysis, based only on the given dataset X = [x_{1}, ⋯, x_{N}], without any additional prior knowledge, such as the labeling of the data samples. We will now consider a few algorithms for clustering analysis.

14.1 K-means clustering

As suggested by the name of K-means clustering, in this method a set of K mean vectors m_{1}, ⋯, m_{K} are used to represent the clusters in the feature space, based on the assumption that there exist a set of K clusters in the dataset.

The K-means clustering algorithm can be formulated as an optimization problem to minimize an objective function

J = ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk}||x_{n} − m_{k}||^{2}  (14.1)

where

P_{nk} = \{ \begin{matrix}1 & if k = arg min_{l} ||x_{n} − m_{l}|| \\ 0 & otherwise \end{matrix}  (14.2)

indicates that x_{n} is assigned to the kth cluster C_{k} if its distance to the mean m_{k} is minimum. To minimize the objective function J with respective to m_{k}, we set its derivative with respect to each m_{k} to zero:

\frac{d}{dm_{k}} J = 2 ∑_{n=1}^{N} P_{nk}||x_{n} − m_{k}|| = 0 k = 1, ⋯, K  (14.3)

and solve the resulting equation for m_{k}:

m_{k} = \frac{∑_{n=1}^{N} P_{nk}x_{n}}{∑_{n=1}^{N} P_{nk}} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x (k = 1, ⋯, K)  (14.4)

where N_{k} = ∑_{n=1}^{N} P_{nk} is the number of all samples assigned to cluster C_{k}, and the summation is over all samples assigned to C_{k}.

As P_{nk} depends on m_{k} while in turn m_{k} also depends on P_{nk}, the steps above need to be carried out iteratively, starting with K randomly initialized mean vectors which are revised iteratively until convergence. Here are steps of the process:

1. Initialize randomly the mean vectors for the K clusters, such as any K samples of the dataset: m_{1}^{(0)}, m_{2}^{(0)}, ...., m_{K}^{(0)}, set iteration index to zero l = 1;

2. Assign every sample x ∈ X in the dataset to one of the K clusters according to its distance to the corresponding mean vector:

if ||x − m_{k}^{(l)}||^{2} = min_{1≤l≤K} ||x − m_{l}^{(l)}||^{2}, then x ∈ C_{k}^{(l)}  (14.5)

where C_{k}^{(l)} denotes the kth cluster with mean vector m_{k}^{(l)} in the lth iteration;

3. Update the mean vectors to get the new mean m_{k}^{(l+1)} so that the objective function J given above, i.e., the sum of the distances squared from all x ∈ C_{k} to m_{k}^{l+1} is minimized:

m_{k}^{(l+1)} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x, (k = 1, ⋯, K)  (14.6)

4. Terminate if the algorithm has converged:

m_{k}^{(l+1)} = m_{k}^{(l)} (k = 1, ⋯, K)  (14.7)

Otherwise, l ← l + 1, go back to Step 2.

This method is simple and effective, but it has the main drawback that the number of clusters K needs to be estimated based on some prior knowledge, and it stays fixed through out the clustering process, even it may turn out later that the dataset may be better fit with more or fewer clusters. One way to resolve this is to carry out the algorithm multiple times with different K, and then evaluate each result based on the objective function J, or some other separability criteria, such as tr(S_{T}^{−1}S_{B}).

The Matlab code for the iteration loop of the algorithm is listed below, where Mold and Mnew are respectively the K mean vectors before and after each modification. The iteration terminates when the mean vectors no longer change.

    Mnew=X(:,randi(N,1,K));          % use any K random samples as initial means
    er=inf;
    while er>0                       % main iteration
        Mold=Mnew;

        Mnew=zeros(d,K);             % initialize new means
        Number=zeros(1,K);
        for i=1:N                    % for all N samples
            x=X(:,i);
            dmin=inf;
            for k=1:K                % for all K clusters
                d=norm(x-Mold(:,k));
                if d<dmin
                    dmin=d;  j=k;
                end
            end
            Number(j)=Number(j)+1;
            Mnew(:,j)=Mnew(:,j)+x;
        end
        for k=1:K
            if Number(k)>0
                Mnew(:,k)=Mnew(:,k)/Number(k);
            end
        end
        er=norm(Mnew-Mold);          % terminate if means no longer change
    end

Example 1

The K-means algorithm is applied to a simulated dataset in 3-D space with C = 4 clusters. The results are shown in the figure below, where the three panels show the results corresponding to K = C − 1 (left), K = C (middle), and K = C + 1 (right). The initial positions of the K means are marked by the black squares, while their subsequent positions through out the iteration are marked by smaller dots. The iteration terminates once the means have moved to the centers of the clusters and no longer change positions.

The clustering results corresponding to K = 3, 4, 5 can be evaluated by the separability tr(S_{T}^{−1}S_{B}), the K intra-cluster istance tr(Σ_{k}) (k = 1, ⋯, K) of the resulting clusters:

K=C-1=3  K=C=4  K=C+1=5
Separability  1.76  2.56  2.58
Intra-cluster distance 9.1 44.3 11.8 10.8 12.7 11.1 9.1 10.8 9.1 11.1 8.9 9.4

(14.8)

and the K(K − 1)/2 inter-cluster (Bhattacharyya) distances (between any two of the K clusters):

K=C-1=3 1  2 2 10.9 3 21.9 184.7  K=C=4 1 2 3 2 4.0 3 5.1 1.6 4 2.4 2.3 4.4  K=C+1=5 1  2 3  4 2 4.0 3 5.1 1.6 4 7.0 4.3 0.3 5 17.3 15.5 8.9 11.1

(14.9)

We see that when K = 3 < C = 4, the intra-cluster distance of the 2nd cluster is significantly greater than the other two, indicating the cluster may contain two smaller clusters. Also, when K = 5 > C = 4, the inter-cluster distance between clusters 3 and 4 is significantly smaller than others, indicating the two clusters are too close and can therefore be merged.

While the K-means method is simple and effective, it has the main shortcoming that the number of clusters K needs to be specified, although in practice it is typically unknown a head of time. In this case, one could try different K values and compare the corresponding results in terms of the intra and inter cluster distances, as well as the separabilities of the resulting clusters, as shown in the example above. Moreover, if the intra-cluster distance of a cluster is too large indicating it may contain more than one cluster (e.g., K = 3 in the example), it can be split; on the other hand, if the inter-cluster distance between two clusters is too small (e.g., K = 5 in the example), the two clusters may belong to the same cluster and need to be merged. Following such merging and/or splitting, a few more iterations can be carried out to make sure the final clustering results are optimal. The figure below shows the clustering results of the same dataset above but modified by merging and splliting. The left pannel is for K = 3, but the second cluster (green) is split, while the right pannel is for K = 5, when the 4th (yellow) and 5th (cyan) clusters are merged.

The idea of modifying the clustering results by merging and spliting leads to the algorithm of Iterative Self-Organizing Data Analysis Technique (ISODATA), which allows the number of clusters K to be adjusted automatically during the iteration by merging clusters close to each other and splitting clusters with large intra-cluster distances. However, this algorithm is highly heuristic as the various parameters such the threshold values for merging and splitting need to be specified.

Example 2

The K-means clustering method is applied to the dataset of ten handwritten digits from 0 to 9 used previously. The clustering result is visualized based on the KLT that maps the data samples in the original 256-D space into the 3-D space spanned by the three eigenvectors corresponding to the three greatest eigenvalues of the covariance matrix of the dataset. The ground truth labelings are color coded as shown on the left, while the clustering result is shown on the right. We see that the clustering results match the original data reasonably well.

The clustering result is also shown in the confusion matrix, of which the columns and rows represent respectively the clusters based on the K-means clustering and the ground truth labeling (not used). In other words, the element in the ith row and jth column is the number of samples labeled to belong to the ith class but assigned to the jth cluster.

[ \begin{matrix}1 & 0 & 25 & 165 & 27 & 2 & 3 & 0 & 1 & 0 \\ 0 & 209 & 0 & 0 & 2 & 0 & 13 & 0 & 0 & 0 \\ 2 & 3 & 2 & 0 & 8 & 2 & 14 & 151 & 0 & 42 \\ 103 & 0 & 0 & 1 & 5 & 3 & 83 & 0 & 0 & 29 \\ 0 & 28 & 1 & 0 & 15 & 2 & 62 & 0 & 115 & 1 \\ 44 & 0 & 1 & 1 & 167 & 0 & 8 & 0 & 3 & 0 \\ 0 & 6 & 160 & 14 & 44 & 0 & 0 & 0 & 0 & 0 \\ 1 & 6 & 0 & 0 & 3 & 122 & 74 & 2 & 1 & 15 \\ 82 & 1 & 1 & 1 & 33 & 3 & 47 & 2 & 0 & 54 \\ 4 & 2 & 0 & 1 & 0 & 28 & 119 & 1 & 64 & 5 \end{matrix} ]  (14.10)

All samples are converted from 256-dimensional vector back to 16 × 16 image form and the averages of the samples in each of the ten cluster are shown in the figure below. We see that each of the ten clusters clearly represents one distinct digit.

14.2 Gaussian mixture model

In K-means clustering, each sample point x is assigned to one of the K clusters if it has the minimum Euclidean distance to the mean of the cluster. But here the distribution of the data samples in the cluster represented by the covariance is not taken into consideration.

This issue can be addressed by the method of Gaussian mixture model (GMM) based on the assumption that each of the K clusters {C_{1}, ⋯, C_{K}} in the given dataset can be modeled by a Gaussian distribution N (x; m_{k}, Σ_{k}), (k = 1, ⋯, K) in terms of both the mean vector m_{k} and covariance matrix Σ_{k}. The overall distribution of the entire dataset X = [x_{1}, ⋯, x_{N}] is the weighted sum of the K Gaussian distributions:

p(x) = ∑_{k=1}^{K} P_{k}N (x; m_{k}, Σ_{k}) = ∑_{k=1}^{K} P_{k} [\frac{1}{(2π)^{d/2}|Σ_{k}|^{1/2}} exp (−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}))]

(14.11)

where P_{k} is the weight for the kth Gaussian N (x; m_{k}, Σ_{k}), satisfying

∫_{−∞}^{∞} p(x) dx = ∑_{k=1}^{K} P_{k} ∫_{−∞}^{∞} N (x; m_{k}, Σ_{k}) dx = ∑_{k=1}^{K} P_{k} = 1  (14.12)

The GMM clustering is carried out by the method of expectation maximization (EM), by which all model parameters denoted by θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} are estimated based on the given dataset, and then each sample in the dataset is assigned to one of the clusters.

We note that the GMM model in Eq. (14.11) is actually the same as Eq. (12.11) in the naive Bayes classification. These two methods are similar in the sense that each cluster or classe C_{k} is modeled by a Gaussian N (x, m_{k}, Σ_{k}), weighted by P_{k}, and the model parameters m_{k} and Σ_{k}, as well as P_{k}, are to be estimated based on the given dataset. However, the two methods are different in that the dataset X in the supervised naive Bayes method is labeled by y, while in GMM such a labeling vector is not available. Instead, here we will introduce a latent or hidden variable z = [z_{1}, ⋯, z_{K}]^{T} as the cluster labeling of the samples in the given dataset X. This latent variable z is binary vector, of which each component is a binary random variable z_{k} ∈ {0, 1}. Only one of these K components is 1, e.g., z_{k} = 1, indicating a sample x in the dataset belongs to the kth cluster C_{k}, while all others are 0, i.e., these K binary variables add up to 1, ∑_{k=1}^{K} z_{k} = 1.

We further introduce the following probabilities for each of the K clusters C_{k} (k = 1, ⋯, K):

• The prior probability for any unobserved data sample x ∈ X in the dataset to belong to C_{k}, represented by z_{k} = 1:

P (z_{k} = 1) = P_{k} (k = 1, ⋯, K) satisfying ∑_{k=1}^{K} P_{k} = 1 (14.13)

These K prior probabilities add up to 1, i.e, the K events z_{k} = 1 (k = 1, ⋯, K) are mutually exclusive and complementary, as any sample x belongs to one and only one of the K clusters.

• The probability distribution of all samples in C_{k}, assumed to be a Gaussian:

p(x|z_{k} = 1, θ) = N (x; m_{k}, Σ_{k})  (14.14)

• The joint distribution of any x and z_{k} = 1:

p(x, z_{k} = 1|θ) = p(x|z_{k} = 1, θ) P (z_{k} = 1) = P_{k}N (x; m_{k}, Σ_{k}) (14.15)

Marginalizing this joint probability over the latent variable z_{k}, we get the Gaussian mixture model, the distribution p(x) of any sample x regardless to which cluster it belongs:

p(x|θ) = ∑_{k=1}^{K} p(x, z_{k} = 1|θ) = ∑_{k=1}^{K} p(x|z_{k} = 1, θ) P (z_{k} = 1) = ∑_{k=1}^{K} P_{k}N (x; m_{k}, Σ_{k})

(14.16)

Note that Eqs. (14.13) through (14.16) are the same as Eqs. (12.8) through (12.11) in the naive Bayes classifier, respectively.

All such probabilities defined for z_{k} = 1 can be generalized to z = [z_{1}, ⋯, z_{K}]^{T} for all K clusters:

p(z|θ) = ∏_{k=1}^{K} P_{k}^{z_{k}}  (14.17)

p(x|z, θ) = ∏_{k=1}^{K} N (x, m_{k}, Σ_{k})^{z_{k}}  (14.18)

p(x, z|θ) = p(z) p(x|z, θ) = ∏_{k=1}^{K} (P_{k}N (x, m_{k}, Σ_{k}))^{z_{k}}  (14.19)

Given the dataset X = [x_{1}, ⋯, x_{N}] containing N i.i.d. samples, we introduce N corresponding latent variables in Z = [z_{1}, ⋯, z_{N}], of which the nth column z_{n} = [z_{n1}, ⋯, z_{nK}]^{T} is the labeling vector of x_{n}, i.e., x_{n} belongs to C_{k} if z_{nk} = 1 (while z_{nl} = 0 for all l ≠ k). Note that here Z = [z_{1}, ⋯, z_{N}] is defined in the same way as Y = [y_{1}, ⋯, y_{N}] in softmax regression (Section 6.2), both as the labeling of X, with the only difference that Y is provided in the training data available for a supervised method, but here Z is a latent variable not part of the data provided for unsupervised clustering. Now we have

p(x_{n}, z_{n}|θ) = ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}} ,  (n = 1, ⋯, N) (14.20)

The likelihood function of the GMM model parameters θ to be estimated can be expressed as:

L(θ|X, Z) = p(X, Z|θ) = p ([x_{1}, ⋯, x_{N}], [z_{1}, ⋯, z_{N}]|m_{k}, Σ_{k}, P_{k}(k = 1, ⋯, K))
= ∏_{n=1}^{N} p(x_{n}, z_{n}|θ) = ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}}  (14.21)

and the log likelihood is:

l(θ|X, Z) = log L(θ|X, Z) = log p(X, Z|θ) = log [∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}}]

= ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]  (14.22)

Similar to the method of maximum likelihodd estimation (MLE) which finds the model parameters in θ as those that maximize the likelihood function L(θ) or its log function log L(θ), here we find the model parameters in θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} as those that maximize the expectation of the log likelihood function above with respect to the latent variables in Z in the following two iterative steps of the EM method:

• E-step: Find the expectation of the log likelihood function.

We first find the posterior probability for an observed x_{n} to belong to cluster C_{k}, indicated by z_{nk} = 1 (and z_{nl} = 0 for all l ≠ k):

P_{nk} = P (z_{nk} = 1|x_{n}, θ) = \frac{p(x_{n}, z_{nk} = 1|θ)}{p(x_{n}|θ)} = \frac{P_{k} N (x_{n}; m_{k}Σ_{k})}{∑_{l=1}^{K} P_{l} N (x_{n}; m_{l}Σ_{l})}
(n = 1, ⋯, N; k = 1, ⋯, K)  (14.23)

As x_{n} has to belong to one of the K clusters, these K posterior probabilities add up to 1, same ss the prior probabilities P (z_{k} = 1) = P_{k}:

∑_{k=1}^{K} P (z_{k} = 1) = ∑_{k=1}^{K} P_{k} = 1,  ∑_{k=1}^{K} P (z_{nk} = 1|x_{n}, θ) = ∑_{k=1}^{K} P_{nk} = 1

(14.24)

This posterior probability P_{nk} = p(z_{nk} = 1|x_{n}, θ) can be considered as the responsibility cluster C_{k} takes for x_{n}. Once we have available all parameters on the right-hand side of Eq. (14.23) for P_{nk}, it is used to assign each sample x_{n} to a cluster C_{k} with the maximun responsibility P_{nk} = max_{l} P_{nl}.

We note that the definition of P_{nk} is similar to the softmax function s_{nk} with the only difference that here the weighted Gaussian function are used instead of the softmax function used in softmax regression.

We also note that the posterior probability P_{nk} defined above represents a soft decision, in the sense that it is possible for any x_{n} to belong to any C_{k} with probability 0 < P_{nk} < 1 for all k = 1, ⋯, K, instead of a hard decision, in the sense that x_{n} belongs to only one specific C_{k} with P_{nk} = 1, while P_{nl} = 0 for all l ≠ k, as in K-means clustering.

We can now find the expectation of the log likelihood with respect to the latent variables in Z:

E_{Z} (log L(θ|X, Z)) = E_{Z} [∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]]

= ∑_{n=1}^{N} ∑_{k=1}^{K} E_{Z}(z_{nk}) [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]

= ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] (14.25)

The last equality is due to the following fact:

E_{z}(z_{nk}) = 1 P (z_{nk} = 1|x_{n}) + 0 P (z_{nk} = 0|x_{n}) = P (z_{nk} = 1|x_{n}) = P_{nk}

(14.26)

• M-step: Find the optimal model parameters that maximize the expectation

of the log likelihood.

We first set to zero the derivatives of the expectation of the log likelihood with respect to each of the parameters in θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)}, and then solve the resulting equations to get the optimal parameters.

– Find P_{k}: As all K of P_{k}’s need to satisfy the constraint ∑_{k=1}^{K} P_{k} = 1, we first construct the Lagrangian function composed of an extra Lagrange multiplier term for the constraint as well as the log likelihood as the objective function:

L(θ, λ) = ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] + λ (∑_{k=1}^{K} P_{k} − 1)

(14.27)

and set it derivative with respect to P_{k} to zero:

\frac{∂}{∂P_{k}} L(θ, λ) = \frac{∂}{∂P_{k}} [∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] + λ (∑_{k=1}^{K} P_{k} − 1)]

= ∑_{n=1}^{N} P_{nk} \frac{1}{P_{k}} + λ = 0  (14.28)

Multiplying both sides by P_{k}, we get

∑_{n=1}^{N} P_{nk} + P_{k}λ = N_{k} + P_{k}λ = 0  (14.29)

where we have defined

N_{k} = ∑_{n=1}^{N} P_{nk}  (14.30)

that satisfies

∑_{k=1}^{K} N_{k} = ∑_{k=1}^{K} (∑_{n=1}^{N} P_{nk}) = ∑_{n=1}^{N} (∑_{k=1}^{K} P_{nk}) = ∑_{n=1}^{N} 1 = N (14.31)

Summing Eq. (14.29) over all k = 1, ⋯, K, we get

∑_{k=1}^{K} (N_{k} + P_{k}λ) = ∑_{k=1}^{K} N_{k} + λ (∑_{k=1}^{K} P_{k}) = N + λ = 0 (14.32)

Substituting λ = −N back into Eq. (14.29), we get the expression for the prior

p(z_{k} = 1) = P_{k} = \frac{N_{k}}{N} = \frac{1}{N} ∑_{n=1}^{N} P_{nk}  (14.33)

This is actually the same as the prior P_{k} in Eq. (12.8) in naive Bayes classification, but here N_{k} defined in Eq. (14.30) is the sum of the probabilities for all N data samples to belong to C_{k}, instead of the number of data samples in C_{k} (unknown in this unsupervised case).

– Find m_{k}:

\frac{∂}{∂m_{k}} E_{Z}(log L(θ|X, Z)) = \frac{∂}{∂m_{k}} [∑_{n=1}^{N} ∑_{l=1}^{K} P_{nl} [log P_{l} + log N (x_{n}, m_{l}, Σ_{l})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} log N (x_{n}, m_{k}, Σ_{k})

=^{∗} ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} [−\frac{1}{2}(x_{n} − m_{k})^{T} Σ_{k}^{−1}(x_{n} − m_{k})]

= \frac{1}{2} ∑_{n=1}^{N} P_{nk}Σ_{k}^{−1}(x_{n} − m_{k}) = 0  (14.34)

Note that here =^{∗} indicates that we have neglected the scaling coefficient (2π)^{−d/2} |Σ_{k}|^{−1/2} of the Gaussian, which is independent of m_{k}. Multiplying 2Σ_{k} on both sides, we get

∑_{n=1}^{N} P_{nk}(x_{n}−m_{k}) = ∑_{n=1}^{N} P_{nk}x_{n}−∑_{n=1}^{N} P_{nk}m_{k} = ∑_{n=1}^{N} P_{nk}x_{n}−N_{k}m_{k} = 0

(14.35)

Solving for m_{k} we get

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}  (14.36)

– Find Σ_{k}:

\frac{∂}{∂Σ_{k}} E_{Z}(log L(θ|X, Z)) = \frac{∂}{∂Σ_{k}} [∑_{n=1}^{N} ∑_{l=1}^{K} P_{nl} [log P_{l} + log N (x_{n}, m_{l}, Σ_{l})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂Σ_{k}} log N (x_{n}, m_{k}, Σ_{k})

= −\frac{1}{2} ∑_{n=1}^{N} P_{nk} [\frac{∂}{∂Σ_{k}} log |Σ_{k}| + \frac{∂}{∂Σ_{k}} (x_{n} − m_{k})^{T} Σ_{k}^{−1}(x_{n} − m_{k})]

= −\frac{1}{2} ∑_{n=1}^{N} P_{nk} [Σ_{k}^{−1} − Σ_{k}^{−1}(x_{n} − m_{k})(x_{n} − m_{k})^{T} Σ_{k}^{−1}] = 0 (14.37)

Here we have used the following facts (for more details see Matrix

Cookbook https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)

\frac{d}{dA} log |A| = (A^{−1})^{T} , \frac{d}{dA} (a^{T} A^{−1}b) = −(A^{−1})^{T} ab^{T} (A^{−1})^{T}

(14.38)

Pre and post multiplying Σ_{k} on both sides of the equation above we get

∑_{n=1}^{N} P_{nk} (Σ_{k} − (x_{n} − m_{k})(x_{n} − m_{k})^{T} ) = 0  (14.39)

Solving for Σ_{k}, we get

Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}(x_{n} − m_{k})(x_{n} − m_{k})^{T}  (14.40)

We note that all model parameters in θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} found respectively in Eqs. (14.33), (14.36), and (14.40) in the M-step depend on P_{nk} in Eq. (14.23) in the E-step, which in turn is a function of these parameters, i.e., the E-step and M-step need to be carried out in an alternative and iterative fashion from some initial values of the parameters until convergence.

In summary, here is the EM clustering algorithm based on Gaussian mixture model:

1. Initialize means m_{k}, covariance Σ_{k} and coefficient P_{k}.

2. The E-step:

Find the responsibility P_{nk} for all N data points and all K clusters and then N_{k}:

P_{nk} = P (r_{k} = 1|x_{n}) = \frac{P_{k} N (x_{n}; m_{k}, Σ_{k})}{∑_{l=1}^{K} P_{l} N (x_{n}; m_{l}, Σ_{l})},  N_{k} = ∑_{n=1}^{N} P_{nk} (14.41)

3. The M-step:

Recalculate the parameters that maximize the likelihood function:

P_{k} = \frac{N_{k}}{N}
m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}

Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}(x_{n} − m_{k})(x_{n} − m_{k})^{T}  (14.42)

4. If the parameters or the log likelihood function have not converged, go back to step 2. Otherwise terminate the iteration. The probability for each sample x_{n} to belong to cluster C_{l} is p_{nl} = P (z_{nl} = 1|x_{n}, θ), and it is therefore assigned to C_{k} if P_{nk} = max_{l} p_{nl}.

We can show that the K-means algorithm is actually a special case of the EM algorithm, when all covariance matrices are the same Σ_{k} = εI, where ε is a scaling factor which appraoches to zero. In this case we have:

p(x|z_{k} = 1, θ) = N (x|m_{k}, εI) = \frac{1}{(2π)^{d/2}ε^{1/2}} exp (−\frac{1}{2ε}||x − m_{k}||^{2}) (14.43)

and the probability for any x_{n} ∈ X to belong to cluster C_{k} is:

P_{nk} = P (z_{k} = 1|x_{n}, θ) = \frac{P_{k}N (x_{n}; m_{l}Σ_{l})}{∑_{l=1}^{K} P_{l}N (x_{n}; m_{l}Σ_{l})} = \frac{P_{k} exp(−||x_{n} − m_{k}||^{2}/2ε)}{∑_{l=1}^{K} P_{l} exp(−||x_{n} − m_{l}||^{2}/2ε)}

(14.44)

When ε → 0, all terms in the denominator approach to zero, but the one with minimum ||x − m_{k}|| approaches to zero most slowly, and becomes the dominant term of the denominator. If the numerator happens to be this term as well, then P_{nk} = 1, otherwise the numerator approaches zero and P_{nk} = 0. Now P_{nk} defined above becomes:

lim_{ε→0} P_{nk} = lim_{ε→0} \frac{P_{k} exp(−||x_{n} − m_{k}||^{2}/2ε)}{∑_{l=1}^{K} P_{l} exp(−||x_{n} − m_{l}||^{2}/2ε)} = \{ \begin{matrix}1 & if ||x_{n} − m_{k}|| = min_{l} ||x_{n} − m_{l}|| \\ 0 & otherwise \end{matrix}

(14.45)

Now the posterior probability 0 < P_{nk} < 1 defined in Eq. (14.23) for a soft decision becomes a binary value P_{nk} ∈ {0, 1} for a hard binary decision to assign x_{n} to C_{k} with the smallest distance. Also N_{k} = ∑_{n=1}^{N} P_{nk} defined in Eq. (14.30) as the sum of the posterior probabilities for all N data points to belong to C_{k} becomes N_{k} as the number of data samples assigned only to C_{k}. In other words, now the probabilistic EM method based on both m and Σ becomes the deterministic K-means method based on m only.

We can also make a comparison between the GMM method for unsupervised clustering and the softmax regression for supervised classification. First, the latent variables Z = [z_{1}, ⋯, z_{N}] in GMM play a similar role as the labeling Y = [y_{1}, ⋯, y_{N}] in softmax regression multi-class classificatioin. However, the difference is that Y is explicitely given in the training set for a supervised classification, while Z is hidden for an unsupervised cllustering analysis. Second, we note that the probability P_{nk} = p(z_{nk} = 1|x_{n}, θ) given in Eq. (14.23) is similar to the softmax function ϕ_{nk} = P (y′ = k|x_{n}) in the softmax method in terms of their form, with the only difference that the Gaussian function is used for GMM while the exponential function is used for solfmax.

Examples

The same dataset is used to test both the K-means and EM clustering methods. The first panel shows 10 iterations of the K-means method, while the second panel shows 16 iterations of the EM method. In both cases, the iteration converges to the last plot. Comparing the two clustering results, we see that the K-means method cannot separate the red and green data points from two different clusters, both normally distributed with similar means but very different covariance matrices, while the blue data points all in the same cluster are separated into two clusters. But the EM method based on the Gaussian mixture model can correctly identified all three clusters.

The two clustering methods are also applied to the Iris dataset, which has three classes each of 50 4-dimensional sample vectors. The PCA method is used to visualize the first two principal compnents, as shown below. Also, as can be seen from their c onfussion matrices, the error rate of the K-means method is 18/150, while that of the EM method is 5/150.

K-means 0 0 50 50 0 0 18 32 0  EM 0 0 50 45 5 0 0 50 0  (14.46)

14.3 Mixture of Bernoulli

If the data are binary, i.e., each data point x is treated as a discrete random variable that takes either of two binary values such as 1 and 0 with probabilities μ and 1 − μ, then the assumption of Gaussian distribution of the dataset is no longer valid and the Gaussian mixture model is not suitable. In this case, the probability mass function (pmf) of the Bernoulli distribution can be used instead:

B(x|μ) = μ^{x}(1 − μ)^{1−x} = \{ \begin{matrix}μ & if x = 1 \\ 1 − μ & if x = 0 \end{matrix}  (14.47)

The mean and variance of x are

E(x) = 1 P (x = 1) + 0 P (x = 0) = 1 μ + 0 (1 − μ) = μ  (14.48)
V ar(x) = E[(x − E(x))^{2}] = E(x^{2}) − E(x)^{2}
= 1^{2} P (x = 1) + 0^{2} P (x = 0) − μ^{2} = μ − μ^{2} = μ(1 − μ) (14.49)

A set of d independent binary variables can be represented as a random vector x = [x_{1}, ⋯, x_{d}]^{T} with mean vector and covariance matrix as shown below:

E(x) = m = [μ_{1}, ⋯, μ_{N}]^{T}  (14.50)
Cov(x) = Σ = diag(μ_{i}(1 − μ_{i})) = [ \begin{matrix}μ_{1}(1 − μ_{1}) & & & & 0 \\ & . & . & . \\ 0 & & & & μ_{d}(1 − μ_{d}) \end{matrix} ] (14.51)

Note that the covariance matrix Σ is solely determined by the means {μ_{1}, ⋯, μ_{N}}. Now we can get the pmf of the a binary random vector x:

B(x|m) = ∏_{i=1}^{d} B(x_{i}|μ_{i}) = ∏_{i=1}^{d} μ_{i}^{x_{i}}(1 − μ_{i})^{1−x_{i}}  (14.52)

and the log pmf:

log B(x|m) = log (∏_{i=1}^{d} B(x_{i}|μ_{i})) = ∑_{i=1}^{d} [x_{i} log μ_{i} + (1 − x_{i}) log(1 − μ_{i})] (14.53)

Similar to the Gaussian mixture model, the Bernoulli mixture model of K multivariate Bernoulli distributions is defined as:

p(x|m_{k}, P_{k}, (k = 1, ⋯, K)) = p(x|θ) = ∑_{k=1}^{K} P_{k}B(x, m_{k}) = ∑_{k=1}^{K} P_{k} ∏_{i=1}^{d} μ_{ki}^{x_{i}}(1−μ_{ki})^{1−x_{i}}

(14.54)

where θ = {m_{k}, P_{k}, (k = 1, ⋯, K)} denotes all parameters of the mixture model to be estimated based on the given dataset, and m_{k} = E_{k}(x) respect to B(x|m_{k}). The mean of this mixture model is

m = E(x) = ∑_{k=1}^{K} P_{k}E_{k}(x) = ∑_{k=1}^{K} P_{k}m_{k}  (14.55)

Also similar to the Gaussian mixture model, we introduce a set of K latent binary random variables z = [z_{1}, ⋯, z_{K}]^{T} with binary conponentsz_{k} ∈ {0, 1} and ∑_{k=1}^{K} z_{k} = 1, and get the prior probability of z, the conditional probability of x given z, and the joint probability of x and z as the following

p(z|θ) = ∏_{k=1}^{K} P_{k}^{z_{k}}  (14.56)

p(x|z, θ) = ∏_{k=1}^{K} B(x, m_{k})^{z_{k}}  (14.57)

p(x, z|θ) = p(z|θ) p(x|z, θ) = ∏_{k=1}^{K} (P_{k} B(x, m_{k}))^{z_{k}}  (14.58)

Given the dataset X = [x_{1}, ⋯, x_{N}] containing N i.i.d. samples, we introduce the corresponding latent variables in Z = [z_{1}, ⋯, z_{N}], of which each z_{n} = [z_{n1}, ⋯, z_{nK}]^{T} is for the labeling of x_{n}. Then we can find the likelihood function of the Bernoulli mixture model parameters θ = {P_{k}, m_{k}, (k = 1, ⋯, K)}:

L(θ|X, Z) = p(X, Z|θ) = p([x_{1}, ⋯, x_{N}], [z_{1}, ⋯, z_{N}]|m_{k}, P_{k}(k = 1, ⋯, K))
= ∏_{n=1}^{N} p(x_{n}, z_{n}|θ) = ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}B(x_{n}, m_{k}))^{z_{nk}} (14.59)

and the log likelihood function:

log L(θ|X, Z) = log p(X, Z|θ) = log ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}B(x_{n}, m_{k}))^{z_{nk}}

= ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log B(x_{n}, m_{k})]  (14.60)

Based on the same EM method used in Gaussian mixture model, we can find the opptimal parameters that maximize the expectation of the log likelihood function in the following two steps:

• E-step: Find the expectation of the likelihood function.

We first find the posterior probability for any sample x_{n} to belong to cluster C_{k}, denoted by P_{nk}:

P_{nk} = P (z_{nk} = 1|x_{n}, θ) = \frac{p(x_{n}, z_{nk} = 1|θ)}{p(x_{n})|θ} = \frac{P_{k} B(x_{n}; m_{k})}{∑_{l=1}^{K} P_{l} B(x_{n}; m_{l})}
(n = 1, ⋯, N; k = 1, ⋯, K)  (14.61)

which is the expectation of z_{nk}:

E(z_{nk}) = 1 P (z_{nk} = 1|x_{n}) + 0 P (z_{nk} = 0|x_{n}) = P (z_{nk} = 1|x_{n}) = P_{nk}

(14.62)

Now we can find the expectation of the log likelihood with respect to the latent variables in Z:

E_{z} (log L(θ|X, Z)) = E_{z} ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log B(x_{n}, m_{k})]

= ∑_{n=1}^{N} ∑_{k=1}^{K} E(z_{nk}) [log P_{k} + log ∏_{i=1}^{d} μ_{ki}^{x_{ni}}(1 − μ_{ki})^{1−x_{ni}}]

= ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]] (14.63)

• M-step: Find the optimal model parameters that maximize the expectation

of the log likelihood function.

We first set to zero the derivatives of the expectation of the log likelihood with respect to each of the parameters in θ = {P_{k}, m_{k} (k = 1, ⋯, K)}, and then solve the resulting equations to get the optimal parameters.

– Find P_{k}: same as in the case of the GMM model:

P_{k} = \frac{N_{k}}{N} = \frac{1}{N} ∑_{n=1}^{N} P_{nk}  (14.64)

– Find m_{k}:

\frac{∂}{∂m_{k}} E_{Z} (log p(X, Z|θ))

= \frac{∂}{∂m_{k}} ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})] = 0 (14.65)

The ith component of the equation is

∑_{n=1}^{N} P_{nk} \frac{d}{dμ_{ki}} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]

= ∑_{n=1}^{N} P_{nk} (\frac{x_{ni}}{μ_{ki}} − \frac{1 − x_{ni}}{1 − μ_{ki}} ) = 0  (14.66)

i.e.,

(1 − μ_{ki}) ∑_{n=1}^{N} P_{nk}x_{ni} = μ_{ki} ∑_{n=1}^{N} P_{nk}(1 − x_{ni}) = μ_{ki}N_{k} − μ_{ki} ∑_{n=1}^{N} P_{nk}x_{ni}

(14.67)

Solving for μ_{ki} we get

μ_{ki} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{ni} (i = 1, ⋯, d)  (14.68)

or, in vector form,

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}  (14.69)

Example:

Clustering results of hand-written digits with K = 10 and K = 12. The mean vectors m_{k} of each of the K clusters are visualized as shown:

14.4 General EM Algorithm

The EM method for the Gaussian mexture model can be generalized to estimate the parameters in θ = {θ_{1}, ⋯, θ_{m}} of a generic probabilistic model p(x, z|θ), where z is a hidden variable that takes any of the K values {z_{1}, ⋯, z_{K}} with the corresponding probabilities P_{1}, ⋯, P_{K}, satisfying ∑_{k=1}^{K} P_{k} = 1.

By marginalizing the hidden variable z we get (similar to how we get Eq. (14.16)):

p(x|θ) = ∑_{k=1}^{K} p(x, z_{k}|θ) = ∑_{k=1}^{K} P (z_{k}|θ) p(x|z_{k}, θ)  (14.70)

Based on the training set X = [x_{1}, ⋯, x_{N}] containing N i.i.d. samples, we can further get the likelihood function of the parameter θ (similar to Eq. (14.21)):

L(θ|X) = ∏_{n=1}^{N} p(x_{n}|θ) = ∏_{n=1}^{N} ∑_{k=1}^{K} P (x_{n}, z_{k}|θ) = ∏_{n=1}^{N} ∑_{k=1}^{K} P_{k} \frac{p(x_{n}, z_{k}|θ)}{P_{k}}

= ∏_{n=1}^{N} E_{z} [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}]  (14.71)

and the log likelihood

l(θ|X) = log L(θ|X) = log ∏_{n=1}^{N} E_{z} [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}]

= ∑_{n=1}^{N} log E_{z} [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}] ≥ ∑_{n=1}^{N} E_{z} log [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}]) (14.72)

The last step is due to Jensen’s inequality (Section A.25) for the concave logarithmic function f(x) = log x.

The final expression is a lower bound of the log likelihood function, which becomes tight if the equality holds, if the expression inside the brackets is not a random variable, but some constant c, i.e., P_{k} = c p(x_{n}, z_{k}|θ) for all k = 1, ⋯, K. This c can be found by

∑_{k=1}^{K} P_{k} = 1 = c ∑_{k=1}^{K} p(x_{n}, z_{k}|θ) i.e., c = \frac{1}{∑_{k=1}^{K} p(x_{n}, z_{k}|θ)}  (14.73)

Now we have

P_{k} = c p(x_{n}, z_{k}|θ) = \frac{p(x_{n}, z_{k}|θ)}{∑_{k=1}^{K} p(x_{n}, z_{k}|θ)} = \frac{p(x_{n}, z_{k}|θ)}{p(x_{n}|θ)} = p(z_{k}|x_{n}, θ) (14.74)

We see that if we set P_{k} to be the same as the posterior probability p(z_{k}|x_{n}, θ) of z_{k}, then the right hand side of Eq. (14.72) is the same as the log likelihood l(θ|X). The optimal parameters in θ that maximize this log likelihood can be found by iterating the following two steps, based on some initial guess of θ:

• The E-step:

P_{k} = p(z_{k}|x_{n}, θ) = \frac{p(x_{n}, z_{k}|θ)}{p(x_{n}|θ)} = \frac{p(x_{n}, z_{k}|θ)}{∑_{k=1}^{K} p(x_{n}, z_{k}|θ)}  (14.75)

We note that this is the same as Eq. (14.23) for the specific Gaussian mixture model.

• The M-step:

θ = arg max_{θ} ∑_{n=1}^{N} E_{z} log [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}] = arg max_{θ} ∑_{n=1}^{N} ∑_{k=1}^{K} P_{k} log [\frac{p(x_{n}, z_{k}|θ)}{P_{k}}]

(14.76)

As P_{k} in the E-step depends on θ, which can be found in the M-step depending on P_{k}, these two steps need to be carried out alternatively and iteratively until the final convergence of θ.

14.5 Linear Models for Binary Classification

We first consider binary classification based on the same linear model y = f(x)+ r = x^{T} w + r used in linear regression considered before. Any test sample x is classified into one of the two classes {C_{0}, C_{1}} depending on whether f(x) is greater or smaller than zero:

if f(x) = w^{T} x \{ \begin{matrix}< 0 \\ > 0 \end{matrix} ,  then x ∈ \{ \begin{matrix}C_{0} \\ C_{1} \end{matrix}  (14.77)

Here w is a model parameter to be determined based on the training set D = {(x_{n}, y_{n})|n = 1, ⋯, N} = {X, y}, where X = [x_{1}, ⋯, x_{N}], y = [y_{1}, ⋯, y_{N}]^{T}, x_{n} ∈ C_{n} if y_{n} = 1 and x_{i} ∈ C_{0} if y_{i} = −1, so that w fits all data points optimally in certain sense.

While in the previously considered least square classification method, we find the optimal w that minimizes the squared error ε(w) = ||r||^{2}, here we find the optimal w based on a probabilistic model. Specifically, we now convert the linear function f(x) = x^{T} w into the probability for x to belong to either class:

p(x ∈ C_{1}|w) = p(y = 1|x, w)
p(x ∈ C_{0}|w) = p(y = −1|x, w) = 1 − p(y = 1|x, w)  (14.78)

This is done by a sigmoid function defined as either a logistic or cumulative Gaussian (error function, erf):

logistic: σ(z) = \frac{1}{1 + e^{−z}} = \frac{e^{z}}{1 + e^{z}}
erf: ϕ(z) = ∫_{−∞}^{z} N (u|0, 1) du = \frac{1}{\sqrt{2π}} ∫_{−∞}^{z} exp (−\frac{1}{2}u^{2}) du(14.79)

In either case, we have

σ(z) = \{ \begin{matrix}0 & z = −∞ \\ 1 & z = ∞ \end{matrix}  and σ(−z) = 1 − σ(z)  (14.80)

Using this sigmoid function, f = x^{T} w in the range of (−∞, ∞) is mapped to σ(f(x)) in the range of (0, 1), which can be used as the probability for x to belong to either class:

p(y = 1|x, w) = σ(x^{T} w) = σ(f)
p(y = −1|x, w) = 1 − p(y = 1|x, w) = 1 − σ(f) = σ(−f)  (14.81)

The two cases can be combined:

p(y|x, w) = σ(y x^{T} w) = σ(yf), or p(y|x, w) = ϕ(yx^{T} w) = ϕ(yf) (14.82)

The binary classification problem can now be treated as a regression problem to find the model parameter w that best fits the data in the training set D = {X, y}. Such a regression problem is called logistic regression if σ(z) is used, or probit regression if ϕ(z) is used.

Same as in the case of Bayesian regression, we assume the prior distribution of w to be a zero-mean Gaussian p(w) = N (0, Σ_{w}), and for simplicity we further assume Σ_{w} = I, and find the likelihood of w based on the linear model applied to the observed data set D = {(x_{n}, y_{n})|n = 1, ⋯, N}:

L(w|D) = p(D|w) = p(y|X, w) = ∏_{n=1}^{N} p(y_{n}|x_{n}, w) = ∏_{n=1}^{N} σ(y_{i}f_{i}) = ∏_{n=1}^{N} σ(y_{n}x_{n}^{T} w)

(14.83)

Note that this likelihood is not Gaussian (as in the case of Eq. (4.63)), because y = ±1 is the binary class labeling of the training samples in X, instead of a continuous function as in the case of regression.

The posterior of w can now be expressed in terms of the prior p(w) and the likelihood p(y|X, w):

p(w|D) = p(w|X, y) = \frac{p(y, w|X)}{p(y)} = \frac{p(y|X, w) p(w)}{p(y|X)} ∝ p(y|X, w) p(w|X)
= ∏_{n=1}^{N} p(y_{n}|x_{n}, w) N (0, Σ_{w}) = ∏_{n=1}^{N} σ(y_{n}f_{n}) \frac{1}{(2π)^{d/2}|Σ_{w}|^{1/2}} exp (−\frac{1}{2}w^{T} Σ_{w}^{−1}w) (14.84)

where the denominator p(y|X) = p(D) is dropped as it is a constant independent of the variable w of interest. We can further find the log posterior denoted by ψ(w):

ψ(w) = log p(w|D) = ∑_{n=1}^{N} log σ(y_{n} f_{n}) − \frac{N}{2} log(2π) − \frac{1}{2} log |Σ_{w}| − \frac{1}{2}w^{T} Σ_{w}^{−1}w

(14.85)

The optimal w that best fits the training set D = {X, y} can now be found as the one that maximizes this posterior p(w|D), or, equivalently, the log posterior ψ(w), by setting the derivative of ψ(w) to zero and solving the resulting equation below by Newton’s method or conjugate gradient ascent method:

g_{ψ}(w) = \frac{d}{dw} (∑_{n=1}^{N} log σ(y_{n}f_{n})) − \frac{d}{dw} (\frac{1}{2}w^{T} Σ_{w}^{−1}w)

= ∑_{n=1}^{N} \frac{d}{dw} (log \frac{1}{1 + e^{−y_{n}x_{n}^{T} w}} ) − Σ_{w}^{−1}w

= ∑_{n=1}^{N} \frac{y_{n}x_{n}}{1 + e^{y_{n}x_{n}^{T} w}} − Σ_{w}^{−1}w = 0  (14.86)

Having found the optimal w, we can classify any test pattern x_{∗} in terms of the posterior of its corresponding labeling y_{∗}:

p(y_{∗} = 1|x_{∗}, w) = σ(x_{∗}^{T} w) = σ(f(x_{∗})) = \frac{1}{1 + exp(x_{∗}^{T} w)}  (14.87)

In a multi-class case with (C_{1}, ⋯, C_{K}), we can still use a vector w_{i} (i = 1, ⋯, K) to represent each class C_{i}, the direction of the class with respect to the origin in the feature space, and the inner product x^{T} w_{i} proportional to the projection of x onto vector w_{i} measures the extent to which x belongs to C_{i}. Similar to the logistic function used in the two-class case, here the soflmax function defined below is used to convert −∞ < x^{T} w_{i} < ∞ (i = 1, ⋯, K) into the probability that x belongs to C_{i}:

p(y ∈ C_{i}|x, W) = \frac{exp(x^{T} w_{i})}{∑_{k=1}^{K} exp(x^{T} w_{k})} = \{ \begin{matrix}0 & x^{T} w_{i} = −∞ \\ 1 & x^{T} w_{i} = ∞ \end{matrix}  (14.88)

where W = [w_{1}, ⋯, w_{K}].

Part V Neural Networks 15 Biologically Inspired Networks

15.1 Biological Inspiration

In machine learning, artificial neural network (ANN) is a category of algorithms inspired by biological neural networks in the brain, and designed to carry out both supervised and unsupervised learning tasks, such as classification and clustering. To understand how such a neural network algorithm works, we first need to consider some basic concepts in biological neural system.

The human brain consists of 10^{11} neurons interconnected through about 10^{14} to 10^{15} synaptic junctions to form millions of neural networks. Hundreds specialized cortical areas are formed based on these networks for different information processing tasks.

Functionally, a neuron consists of the following three parts:

• The cell body or soma containing the nucleus of the cell;

• The dendrites that receive electrochemical stimulations (input impulses) from other neurons and propagate them to the cell body;

• The axon: that conducts the impulses (output) away from the cell body to other cells;

• The synapse is the point at which impulses pass from one cell to another.

The function of a neuron can be modeled mathematically. At each neuron, modeled as a node in the neural network, receives input signals or stimuli x_{1}, ⋯, x_{d} from d other neurons and beomes activated to different extents measured by its activation or net input, the weighted sum of the inputs:

a = ∑_{j=1}^{d} w_{j}x_{j} + b = ∑_{j=0}^{d} w_{j}x_{j} = w^{T} x  (15.1)

where b is the offset or bias, and w_{j} is a weight representing the synaptic connectivity to the jth input node:

w_{j} \{ \begin{matrix}> 0 & excitatory input \\ < 0 & inhibitory input \\ = 0 & no connection \end{matrix}  (15.2)

Also, same as in the case of linear regression, here we have defined x_{0} = 1 and w_{0} = b, so that both the weight and pattern vectors are augmented to become x = [x_{0} = 1, x_{1}, ⋯, x_{d}]^{T} and w = [w_{0} = b, , w_{1}, ⋯, w_{d}]^{T}. The output or response of the neuron is a function of its activation:

y = g(a) = g (∑_{j=1}^{d} w_{j}x_{j} + b) = g(w^{T} x)  (15.3)

Here g(x), the activation function typically takes one of the following forms:

• Liner function:

g(x) = x,  \frac{d g(x)}{dx} = 1  (15.4)

• Logistic sigmoid function:

g(x, a) = \frac{1}{1 + e^{−ax}} = \frac{e^{ax}}{1 + e^{ax}} = \{ \begin{matrix}0 & x = −∞ \\ 1/2 & x = 0 \\ 1 & x = ∞ \end{matrix} , \frac{d g(x)}{dx} = \frac{a e^{−ax}}{(1 + e^{−ax})^{2}}

(15.5)

where parameter a controls the slope of the transition from 0 to 1 around x = 0. Specially

g(x, 0) = \frac{1}{2},  \frac{d g(x)}{dx} = 0  (15.6)
g(x, ∞) = u(x) = \{ \begin{matrix}0 & x < 0 \\ 1 & x > 0 \end{matrix} ,  \frac{d g(x)}{dx} = \{ \begin{matrix}0 & x ≠ 0 \\ ∞ & x = 0 \end{matrix} (15.7)

• Tanh (hyperbolic tangent) function:

g(x, a) = \frac{2}{1 + e^{−ax}} −1 = \frac{e^{ax} − 1}{e^{ax} + 1} = \{ \begin{matrix}−1 & x = −∞ \\ 0 & x = 0 \\ 1 & x = ∞ \end{matrix} , \frac{d g(x)}{dx} = \frac{2a e^{−ax}}{(1 + e^{−ax})^{2}}

(15.8)

• Rectified linear unit (ReLU):

g(x) = max(0, x) = \{ \begin{matrix}0 & x < 0 \\ x & x > 0 \end{matrix} , \frac{d}{dx}g(x) = \{ \begin{matrix}0 & x < 0 \\ 1 & x > 0 \end{matrix} (15.9)

The soft version of ReLU is the softplus function of which the derivatie is the logistic function:

g(x) = log(1 + e^{x}), \frac{d g(x)}{dx} = \frac{e^{x}}{1 + e^{x}} = \frac{1}{1 + e^{−x}}  (15.10)

The function of a neural network can be modeled mathematically as a hierarchical structure shown below containing multiple layers of neurons, called nodes in the context of artificial neural networks:

• The input layer: receives inputs from external sources;

• The output layer: generates output to the external world;

• The hidden layer(s): between of the input and output layers, not visible from outside the network.

The purpose is to train the network according to certain mathematical rules, the learning rules or learning laws, by modifying the weights of a network iteratively based on the inputs (and the desired outputs if the learning is supervised), so that given the input of the network as the stimulus, the network will produce the desired output as the response.

In general, the method of ANN can be used for both supervised and unsupervised learning. For supervised learning, the goal is to model the relationship between certain dependent variable y and independent variable x by a function y = f(x, w), of which the parameter w for all weights in the network is to be estimated based on the training set D = {X, y}. Such a general framework is suitable for both regression and classification. Neural networks can also be used for unsupervised learning based on observed data X without labeling.

Some learning paradigms of the neural networks are listed below, depending on the interpretations of the input and output of the neural network.

1. Pattern Associator

This is the most general form of neural networks that learns and stores the associative relationship between two sets of patterns represented by vectors.

• Training: A set of N pairs of patterns {(x_{n}, y_{n}), n = 1, ⋯, N} is presented to the network which then learns to establish the associative relationship between two sets of patterns:

f : x ∈ R^{d} ⟹ y ∈ R^{m}  (15.11)

• Testing: When a pattern x_{n} in a pair is presented as the input, the network

produces an output pattern y_{n} associated to the output. Human memory is associative in the sense that given one pattern, some as-

sociated pattern(s) may be produced. Examples include: (Evolution, Darwin), (Einstein, E = mc^{2}), (food, sounding bell, salivation).

2. Auto-associator

As a special pattern associator, auto-associator associates a prestored pattern to an incomplete or noisy version of the pattern.

• Training: A set of patterns {x_{1}, ⋯, x_{N}} is presented to the network for it to learn and remember, i.e., the patterns are stored in the network.

• Testing: When an incomplete or noisy version of one of the patterns stored in the network is presented as the input to the network, the original pattern is retrieved by the network as the outpupt.

3. Regression

This is another special kind of pattern associator which takes a vector input x ∈ R^{d} and produces a real value y ∈ R as a multivariable function y = f(x) at its only output node.

• Training: trained by a set of observed data samples, the independent vectors and their corresponding function values {(x_{n}, y_{n}), n = 1, ⋯, N}, the network is model the function.

• Testing: given any vector x, the output value produced by the single output

node is an estimated function value y = f(x).

4. Classification

This is a variation of the pattern associator of which the output patterns are a set of categorical symbols representing different classes {C_{1}, ⋯, C_{K}}, i.e., each input pattern is classified by the network into one of the classes

f : x ∈ R^{d} ⟹ y ∈ {C_{1}, ⋯, C_{K}}  (15.12)

5. Regularity Detector

This is an unsupervised learning process. The network discovers automatically the regularity in the inputs so that similar patterns are automatically detected and grouped together in the same cluster or class.

15.2 Hebbian Learning

Donald Hebb (1949) speculated that “When neuron A repeatedly and persistently takes part in exciting neuron B, the synaptic connection from A to B will be strengthened.” In other words, simultaneous activation of neurons leads to pronounced increases in synaptic strength between them, or “neurons that fire together wire together; neurons that fire out of sync, fail to link”.

For example, the well known classical conditioning (Pavlov, 1927) could be explained by Hebbian learning. Consider the following three patterns:

• Unconditioned stimulus: sight of food F

• Conditioned stimulus: sound of bell B

• Response: salivation S

The unconditioned response is: F → S. Due to the repeated and persistent conditioning process F ∩ B → S, the synaptic connections between patterns B and S are strengthened as both are repeatedly excited simultaneously, i.e., the two patterns become associated, resulting the conditioned response B → S.

Based on this theory of Hebbian learning, the Hebbian network can be considered as a supervised learning method that learns to establish the associative relationship between any pair of two patterns x_{n} and y_{n} in the given dataset {x_{1}, ⋯, x_{N}} and {y_{1}, ⋯, y_{N}}, considered as the training set. This is a 2-layer network with d nodes in the input layer to receive an input pattern x = [x_{1}, ⋯, x_{d}]^{T} and m nodes in the output layer to produce an output y = [y_{1}, ⋯, y_{m}]^{T}. Each output node is fully connected to all d input nodes through its weights:

y_{i} = ∑_{j=1}^{d} w_{ij}x_{j} = w_{i}^{T} x (i = 1, ⋯, m)  (15.13)

where w_{i} = [w_{i1}, ⋯, w_{id}]^{T}, or in matrix form

y = [ y_{1} ; ⋮ ; y_{m} ] = [ w_{1}^{T} ; ⋮ ; w_{m}^{T} ] [ x_{1} ; ⋮ ; x_{d} ] = Wx  (15.14)

where W = [w_{1}, ⋯, w_{m}]^{T} is an m × d matrix.

The Hebbian learning rule is inspired by Hebb’s theory, i.e., when both neurons x_{j} and y_{i} are activated, the synaptic connectivity, here the weight w_{ij}, between them is enhanced:

w_{ij}^{new} = w_{ij}^{old} + η x_{j}y_{i} (i = 1, ⋯, m, j = 1, ⋯, d)  (15.15)

or in matrix form:

W^{new} = W^{old} + η yx^{T}  (15.16)

Here η is the learning rate, a parameter that controls how quickly the weights get modified.

As in all supervised learning, the Hebbian network is first trained and then used for association.

• Training:

For simplicity, we assume all weights are initialized to zero w_{ij} = 0 (i = 1, ⋯, m, j = 1, ⋯, d), and then train the network to find all weights based on all N pattern pairs in the dataset {(x_{n}, y_{n}), n = 1, ⋯, N} based on the learning law:

w_{ij} = ∑_{n=1}^{N} x_{j}^{(n)}y_{i}^{(n)} (i = 1, ⋯, m, j = 1, ⋯, d)  (15.17)

or in matrix form, the weight matrix is the sum of the outer-products of all N pairs of patterns:

W_{m×d} = ∑_{n=1}^{N} y_{n}x_{n}^{T} = ∑_{n=1}^{N} [ y_{1}^{(n)} ; ⋮ ; y_{m}^{(n)} ] [x_{1}^{(n)}, ⋯, x_{d}^{(n)}]  (15.18)

• Association:

When one of the patterns x_{l} is presented to the network, it produces an output:

y = Wx_{l}  (15.19)

which, as shown below, is the same as the associated pattern y_{l}, under the following conditions:

– The N patterns {x_{1}, ⋯, x_{N}} are treated as zero-mean random vectors, and they are assumed to be completely uncorrelated with zero correlation coefficient between any x_{i} and x_{j}:

r_{ij} = \frac{σ_{ij}^{2}}{σ_{i}σ_{j}} = \frac{∑_{k=1}^{d} x_{ki}x_{kj}}{\sqrt{∑_{k=1}^{d} x_{ki}^{2}}\sqrt{∑_{k=1}^{d} x_{kj}^{2}}} = \frac{x_{i}^{T} x_{j}}{||x_{i}|| ||x_{j}||} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}

(15.20)

If we further assume all patterns are normalized with ||x_{n}|| = 1, then we have x_{i}^{T} x_{j} = δ_{ij}

– The number of input nodes is greater than the number of pattern pairs: d ≥ N, i.e., the capacity of the network is large enough for representing N different patterns (as there can be no more than d orthogonal vectors in a d-dimensional space).

Under these conditions, the output of the network as its response to input x_{l} is

y = Wx_{l} = (∑_{n=1}^{N} y_{n}x_{n}^{T} ) x_{l} = y_{l}(x_{l}^{T} x_{l}) + ∑_{n≠l} y_{n}(x_{n}^{T} x_{l}) = y_{l} (15.21)

as x_{n}^{T} x_{l} = 0 for all other terms with n ≠ l. We see that a one-to-one correspondence relationship between x_{n} and y_{n} has been established for all n = 1, ⋯, N. In non-ideal cases when the conditions above are not fully satisfied, the summation term is non-zero and there is an error y − y_{l} ≠ 0.

15.3 Hopfield Network

The Hopfield network is a supervised method, based on the Hebbian learning rule. As a supervised method, the Hopfield network is trained based on a dataset X = [x_{1}, ⋯, x_{K}] containing K data samples each composed of d binary components x = [x_{1}, ⋯, x_{d}]^{T} with x_{i} ∈ {−1, 1}, (i = 1, ⋯, d), representing one of K different of patterns of interest. Once the network is completely trained, the weight matrix W of the network is obtained in which the K patterns are stored. Given an input x, an iterative computation is carried out until convergence, when one of the pre-stored complete pattern that most closely resemble the input x is produced as the output. The Hopfield network can therefore be considered as an auto-associator (a content addressable memory), by which a pre-stored pattern can be retrieved based on its association to the input pattern established by the network.

Structurally, the Hopfield network is a recurrent network, in the sense that the outputs of its single layer of d nodes are fed back to these nodes in an iterative process.

• Training:

The training process is essentially the same as the Hebbian learning, except here the two associated patterns in each pair are the same (selfassociation). The weight matrix of the network is obtained as the sum of the outer-products of the K patterns to be stored:

W_{d×d} = \frac{1}{d} ∑_{k=1}^{K} x_{k}x_{k}^{T} = \frac{1}{d} ∑_{k=1}^{K} [ x_{1}^{(k)} ; ⋮ ; x_{d}^{(k)} ] [x_{1}^{(k)}, ⋯, x_{d}^{(k)}]  (15.22)

The weight connecting node i and node j is defined as

w_{ij} = \frac{1}{d} ∑_{k=1}^{K} x_{i}^{(k)}x_{j}^{(k)} = w_{ji}  (15.23)

We assume no self-connection exists w_{ii} = 0 (i = 1, ⋯, d)

• Autoassociation:

Once the weight matrix W is obtained by the training process, the network can be used as a self-associator. When an input pattern x is presented to the network, the outputs of the network are updated iteratively and asynchronously, one randomly selected node at a time:

x_{i}^{(n+1)} = sgn (∑_{j=1}^{d} w_{ij}x_{j}^{(n)}) = \{ \begin{matrix}+1, & if ∑_{j=1}^{d} w_{ij}x_{j}^{(n)} ≥ 0 \\ −1, & if ∑_{j=1}^{d} w_{ij}x_{j}^{(n)} < 0 \end{matrix} (15.24)

where x_{i}^{(n)} and x_{i}^{(n+1)} are the output x_{i} of the ith node before and after the nth iteration, respectively. As shown below, this iteration will always converge to one of the K pre-stored patterns.

We first define the Energy function of any two nodes x_{i} and x_{j} of x as

e_{ij} = −w_{ij}x_{i}x_{j}  (15.25)

and the total energy of all d nodes in the network as the sum of all pair-wise energies:

E(x) = \frac{1}{2} ∑_{i=1}^{d} ∑_{j=1}^{d} e_{ij} = −\frac{1}{2} ∑_{i=1}^{d} ∑_{j=1}^{d} w_{ij}x_{i}x_{j} = −\frac{1}{2}x^{T} Wx  (15.26)

The interaction between these two nodes is summarized below:

\begin{aligned}& x_{j} & x_{i} & w_{ij} > 0 & w_{ij} < 0 \\ 1 & -1 & -1 & e_{ij} < 0 & e_{ij} > 0 \\ 2 & -1 & 1 & e_{ij} > 0 & e_{ij} < 0 \\ 3 & 1 & -1 & e_{ij} > 0 & e_{ij} < 0 \\ 4 & 1 & 1 & e_{ij} < 0 & e_{ij} > 0 \end{aligned}  (15.27)

We make two observations.

• When the two nodes i and j reinforce each other’s state, e_{ij} < 0:

– If w_{ij} > 0 (in cases 1 and 4), x_{j} = ∓1 tends to keep x_{i} to stay at the same state ∓1 in the iteration.

– If w_{ij} < 0 (in cases 2 and 3), x_{j} = ∓1 tends to keep x_{i} to stay at the

same state ±1.

• When the two nodes i and j change each other’s state, e_{ij} > 0:

– If w_{ij} < 0 (in cases 1 and 4), x_{j} = ∓1 tends to reverse x_{i} from its previous state ∓1 to ±1.

– If w_{ij} > 0 (in cases 2 and 3), x_{j} = ∓1 tends to reverse x_{i} from its previous state ±1 to ∓1.

We note that low energy e_{ij} corresponds to a stable interaction between x_{i} and x_{j}, i.e., they tend to remain unchanged, and high energy corresponds to an unstable interaction, i.e., they tend to change their states. As the result, low total E(x) corresponds to more stable condition of the network, while high E(x) corresponds to less stable condition.

We further show that the total energy E(x) always decreases whenever the state of any node changes. Assume x_{k} has just been changeed, i.e.,x_{k}^{(n+1)} ≠ x_{k}^{(n)} (x_{k} = ±1 but x_{k}^{(n+1)} = ∓1), while all others remain the same x_{l≠k}^{(n+1)} = x_{l≠k}^{(n)}. The energy before x_{k} changes state is

E^{(n)}(x) = −\frac{1}{2} [∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}^{(n)}x_{j}^{(n)} + ∑_{i} w_{ik}x_{i}^{(n)}x_{k}^{(n)} + ∑_{j} w_{kj}x_{k}^{(n)}x_{j}^{(n)}]

= −\frac{1}{2} ∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}x_{j} − ∑_{i} w_{ik}x_{i}^{(n)}x_{k}^{(n)}

and the energy after x_{k} changes state is

E^{(n+1)}(x) = −\frac{1}{2} ∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}x_{j} − ∑_{i} w_{ik}x_{i}^{(n+1)}x_{k}^{(n+1)}  (15.28)

The energy difference is

△E = (x_{k}^{(n)} − x_{k}^{(n+1)}) ∑_{i} w_{ik}x_{i}  (15.29)

Consider two cases:

• Case 1: if x_{k}^{(n)} = −1, but ∑_{i} w_{ik}x_{i} ≥ 0 and x_{k}^{(n+1)} = 1, we have (x_{k}^{(n)} − x_{k}^{(n+1)}) = −2 ≤ 0 and △E ≤ 0.

• Case 2: if x_{k}^{(n)} = 1, but ∑_{i} w_{ik}x_{i} < 0 and x_{k}^{(n+1)} = −1, we have (x_{k}^{(n)} − x_{k}^{(n+1)}) = 2 ≥ 0 and △E ≤ 0.

As in either case, △E(x) ≤ 0 is always true throughout the iteration, we conclude that E(x) will eventually reach one of the minima of the energy landscape, i.e., the iteration will always converge.

We further show that each one of the K pre-stored patterns corresponds to one of the minima of the energy function. The energy function E(x) corresponding to any x can be written as

E(x) = −\frac{1}{2}x^{T} Wx = −\frac{1}{2}x^{T} [\frac{1}{d} ∑_{k=1}^{K} x_{k}x_{k}^{T} ] x = −\frac{1}{2d} ∑_{k=1}^{K} x^{T} x_{k}x_{k}^{T} x = −\frac{1}{2d} ∑_{k=1}^{K}(x^{T} x_{k})^{2}

(15.30)

If x different (ideally orthogonal to) from any of the stored patterns, all K terms of the summation will be small (ideally zero). But if x is the same as any one of the stored patterns, their inner product reaches maximum, causing the total energy to be minimized to reach one of the minima. In other words, the patterns stored in the network correspond to the local minima of the energy function. i.e., these patterns become attractors.

Note that it is possible to have other local minima, called spurious states, which do not represent any of the stored patterns, i.e., the associative memory is not perfect.

16 Perceptron-Based Networks

16.1 Perceptron

The perceptron network (Frank. Rosenblatt, 1958) is a two-layer learning network containing an input layer of d nodes and an output layer of m output node. The perceptron is a supervised method trained by dataset X = [x_{1}, ⋯, x_{N}], of which each sample x = [x_{1}, ⋯, x_{d}]^{T} is labeled by the corresponding component in y = [y_{1}, ⋯, y_{N}]^{T}, indicating to which one of K catagoric classes x_{n} belongs to. The goal of the training process is to determine the weights w = [w_{1}, ⋯, w_{d}]^{T} and bias b so that the output ŷ_{n} = ∑_{i=1}^{d} w_{i}x_{n} + b corresponding to input x_{n} matches its labeling y_{n} for all N samples in the training set in some optimal way. When the perceptron network is fully trained, it can be used to classify any unlabeled sample x into one of the K classes.

We first consider the special case where the output layer has only m = 1 node, and the perceptron is a binary classifier, same as the method of linear regression, and also the initial setup of the support vector machine (SVM), in the sense that all such methods are based on the linear equation f(x) = w^{T} x + b = 0, representing a hyperplane by which the d-dimensional feature space is partitioned into two regions corresponding to two classes C_{+} and C_{−}. When the parameters w and b are determined in the training process, any unlabeled x is classified into either of the two classes depending on whether the function value ŷ = f(x) is greater or smaller than zero:

If f(x) = w^{T} x + b \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then ŷ = sign(f(x)) = \{ \begin{matrix}1 & for x ∈ C_{+} \\ −1 & for x ∈ C_{−} \end{matrix}

(16.1)

If we divide the equation f(x) = w^{T} x + b = 0 by ||w||, we get

p_{w}(x) = \frac{w^{T} x}{||w||} = −\frac{b}{||w||} = b′  (16.2)

where p_{w}(x) = w^{T} x/||w|| is the projection of x onto the normal direction w of the partitioning hyperplane, and b′ = −b/||w|| is the vector from the origin to the hyperplane (|b′| is the distance of the hyperplane to the origin). Now the classification above can be rewritten as

If p_{w}(x) \{ \begin{matrix}> b′ \\ < b′ \end{matrix} , then x ∈ \{ \begin{matrix}C_{+} \\ C_{−} \end{matrix}  (16.3)

We see that in all binary classification methods, an unlabeled sample x is classified into either of the two classes based on the projection p_{w}(x) of x onto w, which is either greater or smaller than the bias b′, depending on whether x is on the positive or negative side of the plane.

In all these binary classification methods the parameters w and b need to be determined based on training set X labeled by y, but they do so differently. While in least squares method and support vector machine these parameters are obtained by respectively the least squared method and quadratic programming, here in the perceptron network these parameters are obtained by iteratively modifying some randomly initialized values to gradually reduce the error or residual, the difference between actual output ŷ_{n} = f(x_{n}) and the ground truth labeling y_{n}, denoted by δ = y_{n} − ŷ_{n}, for all training samples in the dataset. This method is called the δ-rule.

We now consider specifically the training algorithm of the perceptron network as a binary classifier. As always, we redefine the data vector as x = [x_{0} = 1, x_{1}, ⋯, x_{n}]^{T} and the weight vector as w = [w_{0} = b, w_{1}, ⋯, w_{n}]^{T} so that the decision function can be more conveniently written as an inner product f(x) = w^{T} x without the additional bias term.

The randomly initialized weight vector w is iteratively updated based on the following mistake driven δ-rule:

w^{new} = w^{old} + η(y − ŷ) x = w^{old} + η δx  (16.4)

where η > 0 is the step size but here called the learning rate, which is assumed to be 1 in the following for simplicity. We can show that by the iteration above, w is modified in such a way that the error δ is always reduced.

When a training sample x labeled by y ± 1 is presented to the nodes of the input layer of the perceptron, its output ŷ = sign(f(x)) may or may not match the the label y, as shown in the table:

y  f(x) = w^{T} x, ŷ = sign(f(x)) δ = y − ŷ
1 y = 1 f(x) > 0, ŷ = 1  δ = 0
2 y = −1 f(x) > 0, ŷ = 1  δ = −2
3 y = 1 f(x) < 0, ŷ = −1  δ = 2
4 y = −1 f(x) < 0, ŷ = −1  δ = 0

(16.5)

In both the first and last cases, yf(x) > 0, the error is δ = y − ŷ = 0, and the weight vector w^{new} = w^{old} + δx = w^{old} is not modified. But in cases 2 and 3 yf(x) < 0, the error is δ = y − ŷ = 1, the weight vector w is modified in either of the following two ways:

• In case 2, y = −1, but ŷ = 1, then y f(x) < 0 and δ = −2, we have

w^{new} = w^{old} − 2x  (16.6)

When the same x is presented to the network again in the future, the function is smaller than its previous value

f(x) = x^{T} w^{new} = x^{T} w^{old} − 2x^{T} x < x^{T} w^{old}  (16.7)

and the output ŷ is more likely to be the same as the desired y = −1.

• In case 3, y = 1, but ŷ = −1, then y f(x) < 0 and δ = 2, we have

w^{new} = w^{old} + 2x  (16.8)

When the same x is presented again, the function is greater than its previoius value

f(x) = x^{T} w^{new} = x^{T} x^{old} + 2x^{T} x > x^{T} w^{old}  (16.9)

and the output ŷ is more likely to be the same as the desired y = 1.

The update equations in Eq. (16.6) for y = −1 and Eq. (16.8) for y = −1 can be combined to become

w^{new} = w^{old} + yx  (16.10)

where the scaling constant 2 is dropped as it can be absorbed into the learning rate if we let η = 1/2. Now the learning rule can be rewritten as:

If y f(x) \{ \begin{matrix}> 0 \\ < 0 \end{matrix} then \{ \begin{matrix}w^{new} = w^{old} \\ w^{new} = w^{old} + y x \end{matrix}  (16.11)

In summary, the learning law guarantees that the weight vector w is modified in such way that the performance of the network is always improved with reduced error δ = y−ŷ. If C_{+} and C_{−} are linearly saperable, then a perceptron will always produce w in finite number of iterations to saperate them.

This binary classifier with m = 1 output node can now be genrealized to multiclass classifier with m > 1 output nodes, and each of the m weight vectors in W = [w_{1}, ⋯, w_{m}] is modified by the same learning process considered above. The K > 2 classes can be encoded by the m outputs y_{i} ∈ {−1, 1}, (i = 1, ⋯, m) in two different ways. If the one-hot method is used, the m binary output can encode K = m classes, i.e., the kth class is represented by an m-bit output with y_{k} = 1 while all others y_{l} = −1 for l ≠ k. Alternatively, if binary encoding is used, the m outputs can encode as many as K = 2^{m} classes. For example, K = 4 classes can be labeled by 4 binary vector of either 4 or 2 bits:

binary output y_{0} y_{1} y_{2} y_{3}
y_{1}  1 -1 -1 -1
y_{2}  -1 1 -1 -1
y_{3}  -1 -1 1 -1
y_{4}  -1 -1 -1 1

or

binary output y_{0} y_{1} y_{2} y_{3}
y_{1}  -1 -1 1 1
y_{2}  -1 1 -1 1

(16.12)

The outputs of the m output nodes form an m-dimensional binary vector ŷ = [ŷ_{1}, ⋯, ŷ_{m}]^{T}, which is to be compared with the labeling y of the current input x with error δ = ||y − ŷ||. When the training is complete, an unlabeled input x is classified to one of the K classes with a matching label to the perceptron’s output. In the case of one-hot encoding, it is possible for the binary output y to not match any of the K one-hot encoded classes (e.g., ŷ = [−1 1 − 1 1]^{T}. In this case, the input x can be classified to the class corresponding to the node with the greatest output value f(x) = w^{T} x.

The Matlab code for the essential part of the algorithm is listed below. Array X is the dataset contains N training samples, each labeled by one of the components in array y as its class identity. Array W is a (d + 1) × m matrix containing d + 1 dimensional weight vectors each for one of the m output nodes.

    [X Y]=DataOneHot;           % get data
    K=length(unique(Y','rows')) % number of classes
    X=[ones(1,N); X];           % data augmentation
    [d N]=size(X);              % number of dimensions and number of samples
    m=size(Y,1);                % number of output nodes
    W=2*rand(d,m)-1;            % random initialization of weights
    eta=1;
    nt=10^4;                    % maximum number of iteration
    for it=1:nt
        n=randi([1 N]);         % random index
        x=X(:,n);               % pick a training sample x
        y=Y(:,n);               % label of x
        yhat=sign(W'*x);        % binary output
        delta=y-yhat;           % error between desired and actual outputs
        for i=1:m
            W(:,i)=W(:,i)+eta*delta(i)*x;   % update weights for all K output nodes
        end
        if ~mod(it,N)           % test for every epoch
            er=test(X,Y,W);
            if er<10^(-9)
                break
            end
        end
    end

This is the function that test the training set based on estimated weight vectors in W:

function er=test(X,Y,W)          % test based on estimated W,

[d N]=size(X);
Ne=0;                        % number of misclassifications
for n=1:N
    x=X(:,n);
    yhat=sign(W'*x);
    delta=Y(:,n)-yhat;
    if any(delta)            % if misclassification occurs to some output nodes
        Ne=Ne+1;             % update number of misclassifications
    end
end
er=Ne/N;                     % error percentage
end

This is the code that generates the training set labeled by either one-hot or binary encoding method:

function [X,Y]=DataOneHot
d=3;
K=8;
onehot=1;           % onehot=0 for binary encoding
Means=[ -1 -1 -1 -1 1 1 1 1;
    -1 -1 1 1 -1 -1 1 1;
    -1 1 -1 1 -1 1 -1 1];
Nk=50*ones(1,K);
N=sum(Nk);          % total number of samples
X=[];
Y=[];
s=0.4;
s=0.2;
for k=1:K           % for each of the K classes
    Xk=Means(:,k)+s*randn(d,Nk(k));
    if onehot
        Yk=-ones(K,Nk(k));
        Yk(k,:)=1;
    else            % binary encoding
        dy=ceil(log2(K));
        y=2*de2bi(k-1,dy)-1;
        Yk=repmat(y',1,Nk(k));
     end
    X=[X Xk];
    Y=[Y Yk];
end
Visualize(X,Y)
end

Examples

The figure below shows the classification results of a perceptron network with n = 2 input nodes and m = 2 output nodes. The two output nodes encode 2^{2} = 4 classes in a 2-D space. The training set contains 100 samples for each of the four classes labeled by y = [y_{1}, y_{2}]. The two weight vectors w_{1} and w_{2} are initialized randomly. During the training iteration, when δ = y − ŷ ≠ 0, the weight vector for the output node is modified, otherwise, nothing needs to be done. After nine such modifications, the four classes are completely saperated by the two straight lines normal to the two weight vectors. The first panel shows the initial stage, while the subsequent panels show how the weight vectors are modified each time when δ ≠ 0. The darker and bigger dots represent the samples presented to the network when one of the weight vectors is modified.

The figure below shows the classification results of a perceptron of n = 3 input nodes and m = 3 output nodes encoding 2^{3} = 8 classes. After 35 modifications the weight vectors, the perceptron is completely trained to classify all eight classes correctly. The first panel shows the initial stage, the following three panels show the weight vectors, w_{1}, w_{2} and w_{3}, also the normal vectors of the decision planes, from some three different viewing angles. We see that the eight classes are indeed saperated by the three planes normal to the weight vectors.

The main contraint of the perceptron algorithm as a binary classifier that the two classes are linearly separable can be removed by the kernel method, once the algorithm is modified in such a way that all data samples appear in the form of an inner product. Consider first the training process in which the N training samples are repeatedly presented to the network, and the weight vector is modified to become w^{new} = w^{old} + y_{n}x_{n} whenever a sample x_{n} labeled by y_{n} is misclassified with δ = y_{n} − ŷ_{n} ≠ 0. If the weight vector is initialized to zero w = 0, then the weight vector by the updating rule in Eq. (16.10) can be written as a linear combination of the N training samples:

w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}  (16.13)

where α_{n} is the number of times sample x_{n} labeled by y_{n} is misclassified. Upon receiving a new training sample x_{l} labeled by y_{l}, we have

f(x_{l}) = w^{T} x_{l} = ∑_{n=1}^{N} α_{n}y_{n}x_{n}^{T} x_{l}, and ŷ_{l} = sign(f(x_{l}))  (16.14)

and the weight vector w is updated by the delta-rule:

If δ = y_{l} − ŷ_{l} ≠ 0
then w^{new} = w^{old} + y_{l} x_{l} = ∑_{n=1}^{N} α_{n}y_{n}x_{n} + y_{l} x_{l}
i.e. α_{l}^{new} = α_{l}^{old} + 1  (16.15)

We see that only {α_{1}, ⋯, α_{N}} need to be updated during the training process, while the weight vector w in Eq. (16.13) no longer needs to be explicitly calculated. Once the training process is complete, any unlabeled x can be classified into either of the two classes based on {α_{1}, ⋯, α_{N}}:

If w^{T} x = ∑_{n=1}^{N} α_{n}y_{n}x_{n}^{T} x \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then x ∈ \{ \begin{matrix}C_{−} \\ C_{+} \end{matrix}  (16.16)

As all data samples appear in the form of inner product in both the training and testing phase, the kernel method can be applied to replace the inner product x_{n}^{T} x_{m} by a kernel function k(x_{n}, x_{m}). Also, the discussion above for m = 1 output nodes can be generalized to m > 1 output nodes.

Here is the Matlab code segment for the most essential parts of the kernel perceptron algorithm:

[X Y]=Data;                     % get dataset
[d N]=size(X);                  % d: dimension, N: number of training samples
X=[ones(1,N); X];               % augmented data
m=size(Y,1);                    % number of output nodes
A=zeros(m,N);                   % initialize alpha for all m output nodes and N samples
K=Kernel(X,X);                  % get kernel matrix of all N samples
for it=1:nt
    n=randi([1 N]);             % random index
    x=X(:,n);                   % pick a training sample
    y=Y(:,n);                   % and its label
    yhat=sign((A.*Y)*K(:,n));   % get yhat
    delta=Y(:,n)-yhat;          % error between desired and actual output
    for i=1:m                   % for each output node
       if delta(i)~=0           % if a misclassification
           A(i,n)=A(i,n)+1;     % update the corresponding alpha
       end
    end
    if ~mod(it,N)               % test for every epoch
        er=test(X,Y,A);         % percentage of misclassification
        if er<10^(-9)
            break
        end
    end
end
function er=test(X,Y,A)             % function for testing
[d N]=size(X);                  % d: dimension, N: number of training samples
m=size(Y,1);
Ne=0;                           % initialize number of misclassifications
for n=1:N                       % for all N training samples
    x=X(:,n);                   % get the nth sample

        y=Y(:,n);                   % and its label
        f=(A.*Y)*Kernel(X,x);       % f(x)
        yhat=sign(f);               % yhat=sign(f(x))
        if norm(y-yhat)~=0          % misclassification at some output nodes
            Ne=Ne+1;                % increase number of misclassification
        end
    end
    er=Ne/N;                        % percentage of misclassification
end

Example:

The kernel perceptron is applied to the dataset of handwritten digits of K = 10 classes for the digits from 0 to 9 with m = 10 one-hot output nodes, based on the radial basis kernel. The kernel perceptron is trained on half of the data and then tested by the other half of the data. The confussion matricies of the training set by the end of the training process and the test set are both shown below. The error rate of the training set is zeor, i.e., all training samples are correctly classified, while the test error rate is 0.11, i.e., 89% of the test samples are correctly classified.

[ \begin{matrix}116 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 116 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 113 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 122 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 103 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 108 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 109 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 116 \end{matrix} ]  [ \begin{matrix}104 & 7 & 0 & 0 & 0 & 1 & 2 & 1 & 0 & 1 \\ 0 & 103 & 0 & 0 & 0 & 0 & 3 & 1 & 0 & 1 \\ 1 & 0 & 103 & 0 & 0 & 0 & 1 & 1 & 2 & 0 \\ 0 & 0 & 8 & 102 & 0 & 0 & 0 & 2 & 1 & 3 \\ 1 & 1 & 0 & 0 & 103 & 2 & 0 & 0 & 4 & 2 \\ 0 & 0 & 0 & 2 & 0 & 93 & 24 & 0 & 3 & 1 \\ 0 & 1 & 1 & 0 & 0 & 0 & 95 & 4 & 1 & 1 \\ 0 & 0 & 1 & 0 & 0 & 0 & 0 & 104 & 0 & 3 \\ 1 & 0 & 1 & 1 & 3 & 2 & 1 & 3 & 93 & 4 \\ 1 & 2 & 0 & 1 & 7 & 0 & 0 & 3 & 6 & 96 \end{matrix} ]

(16.17)

The constraint of the perceptron algorithm is the requirement that the classes are linear saperabe due to the fact that there is only one level of learning taking place between the output and input layers. This constraint of linear separablity will be removed when multi-layer networks are used, such as the back propagation algorithm to be discussed in the next section, and more generally, the deep learning networks containing a large number of learning layers between the input and output layers.

16.2 Back Propagation

The back propagation network (BP network or BPN) is a supervised laerning algorithm that finds a wide variety of applications in practice. In the most general sense, a BPN can be used as an associator to learn the relationship between two sets of patterns represented in vector forms. Specifically in classification, similar to the perceptron network, the BPN as a classifier is based on the training set X = [x_{1}, ⋯, x_{N}], of which each pattern x_{n}, a d-dimensional vector, is associated with the corresponding pattern, an m-dimensional vector y_{n} in Y = [y_{1}, ⋯, y_{N}], as its class identity labeling, indicating to to which of the K classs {C_{1}, ⋯, C_{K}} pattern x_{i} belongs.

Different from the perceptron in which there is only one level of learning taking place between the output and input layers in terms of the weights, a BPN is a multi-layer (three or more) hierarchical structure composed of the input, hidden, and output layers, in which learning takes place in multilevels between consecutive layers. Consequently, a BPN can be more flexible and powerful than the two-layer perceptron network. In the following, before we consider the general BPN containg multiple hidden layers, we will first derive the back propagation algorithm for the simplest BPN with only one hidden layer in between the input and output layers.

We assume the input, hidden and output layers contain respectively d, l, and m nodes, each node in the hidden and output layers is fully connected to all nodes in the previous layer. When one of the N training patterns x is presented to the input layer of the BPN, an m-D vector ŷ = f(x, W^{h}, W^{o}) is produced at the output layer as the corresponding response to the input x. Here W^{h} = [w_{1}^{h}, ⋯, w_{l}^{h}] and W^{o} = [w_{1}^{o}, ⋯, w_{m}^{o}] are the function parameters containing the augmented weight vectors for both the hidden and output layers, to be determined in the training process based on the training set X and Y so that its output ŷ_{n} as a function of the current input x_{n} matches the desired output, the labeling y_{n}. Once the BPN is fully trained, any unlabeled pattern x can then be classified into one of the K classes corresponding to the minimum δ = ||y − ŷ||. Note that different from the perceptron network, the output of the m output nodes are in general not binary, although they can still be binary if either onehot or binary encoding is used for class labeling.

Specifically, the training of the BPN is an iteration of the following two-phase process:

• The feedforward pass:

A randomly selected sample x_{n} labeled by y_{n} is presented to the input layer and forwarded through the weighted connections to the hidden layer and then the output layer to produce output ŷ = f(x, W^{h}, W^{o}).

• The backward error backpropagation:

The squared error ε = ||y_{n} − ŷ_{n}||^{2}/2 measuring the difference between the desired output y_{n} and the actual output ŷ_{n} is propagated backward from the output layer through the hidden layer to the input layer, during which the weights of both the output and hidden layers are modified so that ε will be reduced when the same or similar pattern is presented in the future.

This two-phase process is iteratively carried out untill eventually the error is minimized for all training samples and BPN is properly trained.

We now consider the specific computation taking place in both the forward and backward passes.

• The forward pass from input x = [x_{1}, ⋯, x_{d}]^{T} represented to the input layer to the output ŷ_{n} = [y_{1}, ⋯, y_{m}]^{T} produced by the output layer:

– From input layer to hidden layer:

z_{j} = g(a_{j}^{h}) = g (∑_{k=1}^{d} w_{jk}^{h}x_{k} + b_{j}^{h}) = g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) = g(x^{T} w_{j}^{h}) (j = 1, ⋯, l)

(16.18)

where both x = [x_{0} = 1, x_{1}, ⋯, x_{d}]^{T} and w_{j}^{h} = [w_{j0}^{h} = b_{j}^{h}, w_{j1}^{h}, ⋯, w_{jd}^{h}]^{T} are augmented, and a_{j}^{h} = x^{T} w_{j}^{h} is the activation of the jth hidden layer node. These l equations can be expressed in vector form:

z = g (W^{h}x)  (16.19)

where z = [z_{1}, ⋯, z_{l}]^{T}, and W^{h} = [w_{1}^{h}, ⋯, w_{l}^{h}]^{T} is an l × (d + 1) matrix.

– From hidden layer to output layer:

ŷ_{i} = g(a_{i}^{o}) = g (∑_{j=1}^{l} w_{ij}^{o}z_{j} + b_{i}^{o}) = g (∑_{j=0}^{l} w_{ij}^{o}z_{j})

= g (∑_{j=0}^{l} w_{ij}^{o}g (∑_{k=0}^{d} w_{jk}^{h}x_{k})) (i = 1, ⋯, m) (16.20)

which can also be written in vector form as:

ŷ_{i} = g(z^{T} w_{i}^{o})  (16.21)

where both z = [z_{0} = 1, z_{1}, ⋯, z_{l}]^{T} and w_{i}^{o} = [w_{i0}^{o} = b_{i}^{o}, w_{i1}^{o}, ⋯, w_{il}^{o}]^{T} are augmented, and a_{i}^{o} = z^{T} w_{i}^{o} is the activation of the ith output layer node. These m equations can be expressed in vector form:

ŷ = g (W^{o}z)  (16.22)

where ŷ = [ŷ_{1}, ⋯, y_{m}]^{T}, and W^{o} = [w_{1}^{o}, ⋯, w_{m}^{o}]^{T} is an m × (l + 1) matrix.

• The backward error propagation from error ε to the input layer:

The squared error ε = ||y − ŷ||^{2}/2 can be written as a function of the output weights w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) and hidden layer weights w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d):

ε = \frac{1}{2}||y − ŷ||^{2} = \frac{1}{2} ∑_{i=1}^{m}(ŷ_{i} − y_{i})^{2} = \frac{1}{2} ∑_{i=1}^{m}[g(a_{i}^{o}) − y_{i}]^{2} = \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o}z_{j}) − y_{i}]^{2}

= \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o} g(a_{j}^{h})) − y_{i}]^{2} = \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o} g (∑_{k=0}^{d} w_{jk}^{h}x_{k})) − y_{i}]^{2} (16.23)

Similar to the objective function of the ridge regression considered in Section 4.3, an additional regularization term weighted by the weight decay parameter λ can be included in the objective function to penalize large weights and thereby prevent overfitting:

J(W^{h}, W^{o}) = ε + \frac{λ}{2} [||W^{h}||_{2}^{2} + ||W^{o}||_{2}^{2}]  (16.24)

where the two weight matrices in the regularization term should exclude the first column for the bais term b which need not to be forced to be small, and ||W||_{2} = \sqrt{∑_{i} ∑_{j} w_{ij}^{2}} is the Frobenius norm (Section A.17).

The gradient descent method is used to modify first the output layer weights and then the hidden layer weights to iteratively reduce the objective function J in the following steps:

– Find the gradient of J with respect to the output layer weights w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) by the chain rule:

\frac{∂ J}{∂ w_{ij}^{o}} = \frac{∂ ε}{∂ ŷ_{i}} \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ w_{ij}^{o}} +λw_{ij}^{o} = (ŷ_{i}−y_{i}) g′(a_{i}^{o}) z_{j}+λw_{ij}^{o} = −δ_{i} g′(a_{i}^{o}) z_{j}+λw_{ij}^{o}

(16.25)

where δ_{i} = y_{i} − ŷ_{i}.

– Update w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) to reduce J by gradient descent method with learning rate or step size η:

w_{ij}^{o} ⇐ w_{ij}^{o} − η \frac{∂J}{∂w_{ij}^{o}} = w_{ij}^{o} − η (−δ_{i} g′(a_{i}^{o}) z_{j} + λw_{ij}^{o})
= w_{ij}^{o} − η (−d_{i}^{o} z_{j} + λw_{ij}^{o})  (16.26)

where d_{i}^{o} = δ_{i}g′(a_{i}^{o}), or in matrix form:

W^{o} ⇐ W^{o} − η (−d^{o}z^{T} + λW^{o}) = W^{o} + ηd^{o}z^{T} − ηλW^{o} (16.27)

where d^{o} = [d_{1}^{o}, ⋯, d_{m}^{o}]^{T} is the elementwise (Hadamard) product δ ⊙ g′(a^{o}), and d^{o}z^{T} = d^{o} ⊗ z is the outer product of d^{o} and z = [z_{0} = 1, z_{1}, ⋯, z_{l}].

– Find the gradient of J with respect to the hidden layer weights w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d) by the chain rule:

\frac{∂ J}{∂ w_{jk}^{h}} = \frac{∂ ε}{∂ ŷ_{i}} \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ z_{j}} \frac{∂ z_{j}}{∂ a_{j}^{h}} \frac{∂ a_{j}^{h}}{∂ w_{jk}^{h}} + λw_{jk}^{h} = ∑_{i=1}^{m} (ŷ_{i} − y_{i}) \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ z_{j}} \frac{∂ z_{j}}{∂ a_{j}^{h}} \frac{∂ a_{j}^{h}}{∂ w_{jk}^{h}} + λw_{jk}^{h}

= − ∑_{i=1}^{m} δ_{i}g′(a_{i}^{o})w_{ij}^{o} g′(a_{j}^{h})x_{k} + λw_{jk}^{h} = − (∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) g′(a_{j}^{h})x_{k} + λw_{jk}^{h}
= −δ_{j}^{h} g′(a_{j}^{h})x_{k} + λw_{jk}^{h} = −d_{j}^{h}x_{k} + λw_{jk}^{h}  (16.28)

where we have defined

δ_{j}^{h} = ∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}, d_{j}^{h} = δ_{j}^{h}g′(a_{j}^{h}) = (∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) g′(a_{j}^{h}), (j = 1, ⋯, l)

(16.29)

or in matrix form:

[ δ_{1}^{h} ; ⋮ ; δ_{l}^{h} ] = [ \begin{matrix}w_{11}^{o} & ⋯ & w_{1m}^{o} \\ ⋮ & ... & ⋮ \\ w_{l1}^{o} & ⋯ & w_{lm}^{o} \end{matrix} ] [ d_{1} ; ⋮ ; d_{m} ] = W^{o}d^{o}  (16.30)

Here W^{o} is an m × l matrix, the same as that defined above but with the first column of b_{j}’s removed.

– Update w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d) to reduce J by gradient descent method:

w_{jk}^{h} ⇐ w_{jk}^{h} − η \frac{∂J}{∂w_{jk}^{h}} = w_{jk}^{h} − η (−δ_{j}^{h} g′(a_{j}^{h})x_{k} + λw_{jk}^{h})
= w_{jk}^{h} − η (−d_{j}^{h}x_{k} + λw_{jk}^{h})  (16.31)

or in matrix form:

W^{h} ⇐ W^{h}−η (−W^{o}d^{o} ⊙ g′(a^{h})x^{T} + λW^{h}) = W^{h}+ηd^{h}x^{T} −ηλW^{h}

(16.32)

where d^{h} = W^{o}d^{o} ⊙ g′(a^{h}) is the elementwise product of vector W^{o}d^{o} and g′(a^{h}), and d^{h}x^{T} = d^{h} ⊗ x is the outer product of vectors d^{h} and x.

In summary, here are the steps in each iteration:

1. Input a randomly selected pattern [x_{1}, ⋯, x_{d}]^{T}, construct d + 1 dimensional vector x = [1, x_{1}, ⋯, x_{d}]^{T};

2. Compute z = g(W^{h}x), and construct l + 1 dimensional vector z ← [1, z];

3. Compute ŷ = g(W^{o}z);

4. Get elementwise product d^{o} = (y − ŷ) ⊙ g′(a^{o}) = δ ⊙ g′(a^{o});

5. Update output weights W^{o} ⇐ W^{o} + η d^{o}z^{T} − ηλW^{o};

6. Get elementwise product d^{h} = W_{m×l}^{o}d^{o} ⊙g′(a^{h}) = δ^{h} ⊙g′(a^{h}), where W_{m×l}^{o} is the same as W^{o} but with its first column removed.

7. Update hidden weights W^{h} ⇐ W^{h} + η d^{h}x^{T} − ηλW^{h}; 8. Terminate the iteration if the error ε is acceptably small for all of the training

patterns. Otherwise repeat the above with another pattern in the training set.

The Matlab code for the essential part of the BPN algorithm is listed below. Array X contains C classes each with K samples, array Y are the labelings of the C ∗ K training samples, array W contains the N + 1 dimensional weight vectors for the M output nodes. Also, L is the number of hidden nodes, eta is the learning rate between 0 and 1, and tol is the tolerance for the terminination of the learning iteration (e.g., 0.01).

function [Wh, Wo, g]=backPropagate(X,Y)
syms x;
g=1/(1+exp(-x));              % Sigmoid activation function
dg=diff(g);                   % its derivative function
g=matlabFunction(g);
dg=matlabFunction(dg);
[d,N]=size(X);                % number of inputs and number of samples
X=[ones(1,N); X];             % augment X by adding a row of x0=1
Wh=1-2*rand(L,d+1);           % Initialize hidden layer weights
Wo=1-2*rand(M,L+1);           % Initialize output layer weights
er=inf;
while er > tol
    I=randperm(N);            % random permutation of samples
    er=0;
    for n=1:N                 % for all N samples for an epoch
        x=X(:,I(n));          % pick a training sample
        ah=Wh*x;              % activation of hidden layer
        z=[1; g(ah)];         % augment z by adding z0=1
        ao=Wo*z;              % activation to output layer
        yhat=g(ao);           % output of output layer
        delta=Y(:,I(n))-yhat  % delta error
        er=er+norm(delta)/N   % test error
        do=delta.*dg(ao);     % Find d of output layer
        Wo=Wo+eta*do*z';      % update weights for output layer
        dh=(Wo(:,2:L+1)'*do).*dg(ah);   % delta of hidden layer
        Wh=Wh+eta*dh*x';      % update weights for hidden layer
    end
end
end

The training process of BP network can also be considered as a data modeling problem as in regression to fit the given data {(x_{i}, y_{i}), (i = 1, ⋯, N)} by a function with the weights of both the hidden and output layers as the parameters:

y = f(x, W^{h}, W^{o})  (16.33)

The goal is to find the optimal parameters W^{h} and W^{o} that minimize the difference r = y −ŷ between the desired and the actual outputs. The Levenberg-Marquardt algorithm discussed previously can be used to obtain the parameters

(e.g., the Matlab function trainlm http://www.mathworks.com/help/nnet/ref/trainlm.html).

The three-layer BPN containing a single hidden layer discussed above can be easily generalized to a multilayer BPN containing any number of hidden layers (e.g., for a deep learning network) by simply repeating steps 6 and 7 for the single hidden layer in the algorithm list above. We assume in addition to the input layer, there are in total L learning layers including all the hidden layers and the output layer, indexed by l = 1, ⋯, L. Then step 6 above becomes:

d^{(l)} = δ^{(l)} · g′(a^{(l)}) (l = 1, ⋯, L)  (16.34)

where a^{(l)} = (W^{(l)}z^{(l)}) is the activation of all nodes in the lth layer, z^{(l)} = W^{(l−1)}z^{(l−1)} is the input to the lth layer, output from the (l-1)th layer (z^{(1)} = x), and

δ^{(l)} = \{ \begin{matrix}y − ŷ & (l = L) \\ W^{(l)}d^{(l−1)} & (1 ≤ l < L) \end{matrix}  (16.35)

Then the same step 7 for updating the weights can be carried out based on the gradient vector:

d^{(l)} ⊗ z^{(l)} = d^{(l)}(z^{(l)})^{T}  (16.36)

The matlab code segment below is for the BP network of multiple hidden layers, in which L is the total number of learning layers and m(l) is the number of nodes in the lth layer (l = 1, ⋯, L):

    W={1-2*rand(m(1),d+1)};                 % initial weights for first layer
    for l=2:L
        W{l}=1-2*rand(m(l),m(l-1)+1);       % initial weights for all other layers
    end
    er=inf;
    while er > tol
        I=randperm(N);                      % random permutation of samples
        er=0;
        for n=1:N                           % N samples for an epoch
            z={1;X(:,I(n))};                % pick a training sample
            a={W{1}*z{1}};                  % activation of first layer
            for l=2:L                       % the forward pass
                z{l}=[1;g(a{l-1})];         % input to the lth layer
                a{l}=W{l}*z{l};             % activation of the lth layer
            end
            yhat=g(a{L});                   % actual output of last layer
            delta=Y(:,I(n))-yhat;           % delta error
            er=er+norm(delta)/N;            % test error
            d{L}=delta.*dg(a{L});           % d for output layer

            W{L}=W{L}+eta*d{L}*z{L}';       % upddate weights for output layer
            for l=L-1:-1:1                  % the backward pass
                d{l}=(W{l+1}(:,2:end)'*d{l+1}).*dg(a{l});  % d for hidden layers
                W{l}=W{l}+eta*d{l}*z{l}';   % upddate  weights for hidden layers
            end
        end
    end

Example 1: The classification results of two previously used 2-D data sets are shown below. The error rates are respectively 13% and 11.5%, and the confusion matrices are:

[ \begin{matrix}185 & 14 & 1 \\ 5 & 181 & 14 \\ 11 & 33 & 156 \end{matrix} ]  [ \begin{matrix}176 & 24 \\ 22 & 178 \end{matrix} ]  (16.37)

Example 2:

The back propagation network applied to the classification of the dataset of handwritten digits from 0 to 9 used previously. Out of the N = 2240 samples in the dataset, half is used for training while the other half for testing. Shown below are the confusion matrices of both the training (left) and testing (right) phases. Out of the 1120 samples for training 27 are misclassified (2.4%), and out of the 1120 samples for testing 74 are misclassified (6.6%).

[ \begin{matrix}113 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\ 0 & 107 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 1 & 106 & 1 & 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 4 & 113 & 0 & 0 & 0 & 0 & 0 & 3 \\ 0 & 0 & 0 & 0 & 113 & 8 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 1 & 120 & 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 102 & 0 & 0 & 0 \\ 0 & 1 & 2 & 0 & 0 & 1 & 0 & 103 & 0 & 1 \\ 0 & 1 & 0 & 1 & 2 & 0 & 0 & 0 & 104 & 1 \\ 1 & 1 & 0 & 0 & 2 & 0 & 0 & 0 & 0 & 112 \end{matrix} ] ,  [ \begin{matrix}100 & 1 & 0 & 0 & 0 & 4 & 3 & 0 & 0 & 0 \\ 0 & 111 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 1 \\ 1 & 0 & 113 & 1 & 0 & 0 & 0 & 1 & 0 & 0 \\ 2 & 1 & 1 & 100 & 0 & 0 & 0 & 1 & 0 & 3 \\ 0 & 1 & 0 & 0 & 105 & 0 & 0 & 0 & 2 & 3 \\ 1 & 0 & 0 & 0 & 3 & 95 & 0 & 0 & 2 & 0 \\ 0 & 2 & 1 & 0 & 0 & 0 & 118 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1 & 1 & 0 & 0 & 105 & 1 & 5 \\ 1 & 1 & 2 & 1 & 1 & 5 & 0 & 2 & 97 & 5 \\ 0 & 2 & 0 & 0 & 4 & 0 & 0 & 0 & 0 & 102 \end{matrix} ]

(16.38)

Before leaving this section, we further make the following comments regarding how the back probagation network, and in genral, multilayer feed forward network, is closely related to the general method of regression discussed previously.

• The computational model for a single neuron in Eq. (15.3) based on the logistic activation function in Eq. (15.5) is actaully the same as Eq. (6.6) in logistic regression in Section 6.1:

g (∑_{j=1}^{d} w_{j}x_{j} + b) = g(w^{T} x) = \frac{1}{1 + e^{−w^{T} x}}  (16.39)

treated as the conditional probability p(ŷ = 1|x, w). Both methods are to determine the model parameter w, but based on different methods.

• A supervised learning neural network can be considered as a function ŷ = f(x, w) as a model of the training dataset D = {X, y}, while the parameter w can be found by solving an optimization problem (Chapter 4), by either the least squares method that minimizes ε = ||y − ŷ||^{2}, or the maximum likelihood method that maximizes L(w) = P (y|X, w), equivalent to minimizing −l(w) = − log L(w), which can be interpreted as the cross entropy between the model distribution based on D and the unknown ground true distribution, as discussed in Section 4.1.

• As a neural network can be considered as a model for the function y = f(x) relating y to x in the given dataset D, a fundamontal question to ask is whether such a network is able to model any function. This qustion is answered by the universalk approximation theorem, stating that a feedforward network with arbitrary width (number of nodes in a hidden layer) or depth (number of hidden layers) can represent any function with arbitrary accuracy.

While this theorem is formally proven, it can also be understood intuitively. Recall the method of linear regression based on basis functions (Chapter 4), by which a function is approximated by a repression function as a linear combination of K basis functions (Eq. (4.59)):

ŷ_{m} = ∑_{k=1}^{K} w_{mk}^{o} φ_{k}(x) = ∑_{k=1}^{K} w_{mk}^{o} g (∑_{i=0}^{d} w_{ki}^{h}x_{i})  (16.40)

which in the same form as in Eq. (16.20) for the three-layer feed forward network discussed above, with a linear activation function for the output layer. We see the the output of the hidden layer nodes are the basis functions, and the output of an output layer node is a linear combination of such basis functions. As shown in Example 3 there, a given function can be approximated with progressively higher accuracy as more basis functions, i.e., more hidden layer nodes, are used, if the width of the network is unbounded.

If the width of the network is bounded but its depth is unbounded, we can increase the number of layers of the network so that in the equation above we can have as many levels of summations as needed, i.e., the total number of terms in these summations can increased without a bound, then by the same argument, such a network can again approximate a given function with arbitrary accuracy.

16.3 Autoencoder

Autoencoder is a neural network method that learns to represent the patterns in a given dataset in such a way that both the dimensionality and the noise can be reduced. An autoencoder is similar to a back propogation network in that it iteratively updates the weights in all hidden and output layers by back propagation so that eventually its outputs are as desired. But different from a back propogation network that learns to associate a pattern y_{i} to another pattern x_{i} as its labeling, the autoencoder is an unsupervised learning method based on unlabeled data, and it learns to associate each pattern x_{i} to itself.

The purpose of the autoencoder is not to simply copy the data, instead, by imposing constraints such as sparsity to the hidden layer(s) in terms of the number of active nodes, the number of nodes in the hidden layer is much smaller than those in the input and output layers, and the hidden layer becomes the bottleneck of the multi-layer network, and can be considered as the code by which the information contained in the information presented by the input layer and reproduced by the output layer is encoded by the hidden layer nodes in a lower dimensional space.

The computation taking place in an autoencoder can therefore be considered as an encoding/decoding process. The encoder is composed of the input layer and the hidden layers, and the decoder is composed of the hidden layers and the output layer. Due to the limited number of nodes in the hidden layers, the network is forced to represent the data in reduced data size for data compression, if the components of x in the dataset are correlated, and to discover certain potential structures in the dataset.

We see that an autoencoder is similar to the method of PCA based on the KLT in that both can extract essential information from the data and compress the data without losing much information. However, the PCA does so in a linear manner as the principal components are linear combinations of the input signal components; while in autoencoder, the components generated by the hidden layer nodes are nonlinear functions of the input signal components due to the activation function. For this reason, an autoencoder can be a more powerful method than PCA, but with a higher computational complexity.

Same as in back propogation, a pattern vector x presented to the input layer is forward propogated first to the hidden layer (same as Eq. (16.19)):

z = g(W^{h}x),  or z_{j} = g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) (j = 1, ⋯, l)  (16.41)

and then further to the output layer (same as Eq. (16.22)):

x̂ = g(W^{o}z)  or x̂_{i} = g (∑_{j=0}^{l} w_{ij}^{o}z_{k}) (i = 1, ⋯, m)  (16.42)

Also same as in back propogation, the weigths in both W^{h} and W^{o} are updated during training in the back propogation process so that the following objective function for the squared error between the input and desired output, the same as the input, is minimized:

ε = \frac{1}{2}||x − x̂||^{2}  (16.43)

Here we assume there is only a single hidden layer for convenicne, but the computation for both the forward and backward passes can be easily extended to multiple layers.

By limiting the number l of the hidden layer to be smaller than that of input and output layer, the dimensionality of the data can be reduced from d for the dimensionality of the input x = [x_{1}, ⋯, x_{d}]^{T} to l, the number of hidden layer nodes, while most of the information in the original data is reserved.

It is interesting to compare the autoencoder and the PCA based on KLT transform in terms of how they map a given data vector x in the original d-dimensional space to a lower-dimensional space. Recall the KLT previously considered:

y = [ y_{1} ; ⋮ ; y_{d} ] = V^{T} x = [  v_{1}^{T} ; ⋮ ; v_{d}^{T}  ] x = [ v_{1}^{T} x ; ⋮ ; v_{d}^{T} x ]  (16.44)

In KLT, the ith component y_{i} = v_{i}^{T} x of y is the inner product of the ith eigenvector v_{i} and the input x, the projection of x onto v_{i} if it is normalized. On the other hand, in a neural network, the inner product of the weight vector w_{i} and the input x is further mapped to z_{i} = g(w_{i}^{T} x). If, specially, the activation function g(u) = u is linear (an identical function), the weight vectors w_{i} play the same role as the eigenvectors v_{i} as the basis vectors that span the space, i.e., autoencoder and PCA are essentially the same. But as in general, the activation function g(u) is nonlinear, the autoencoder is a nonlinear mapping, of which the linear KLT (a rotation in the space) is a special case.

Dimension reduction exemplified by the KLT is effective if the components of the pattern vectors are highly correlated, i.e., some of them may carry similar information and therefore are redundant. However, if this is not the case, i.e., if all components carry their own independent information, dimensionality reduction will inevitably cause significant information loss. Similarly in an autoencoder, if the components of the data points are not mostly independent of each other, the autoencoder may not be able to reproduce them as the output if the number of hidden layer nodes is too small.

Example 1:

As dimension reduction methods, both autoencoder and PCA are applied to two datasets, the 4-D iris dataset of three classes, and a set containing eight classes each in one of the octants of the 3-D space. Then comparison of the two methods is made in terms of the separability in the resulting space of lower dimensions, measured by J_{B/T} = tr (S_{T}^{−1}S_{B}) defined in Eq. (8.24) in Section 8.2, and the classification accuracy (by the naive Bayes method) measured by error rate. Specifically, during the back propagation training of the autoencoder, all samples in each dataset are used in random order as both the input and the desired out for the network gradually learn these patterns.

The figures above show the two datasets (the 3-D 8-class dataset at top, the 4-D 3-class iris dataset at bottom) when mapped to a 2-D space by the autoencoder (left) and PCA (right). We note that the data points are similarly distributed in the 2-D spaces resulted from the two methods, indicating the spaces are spanned by similar basis vectors, i.e., the weight vectors of the two hidden layer nodes are similar to the eigenvectors used in the PCA. However, by visual observation, the class separability in the 2-D space resulting from the autoencoder (left) seems to be a little better than that by the KLT (right), which is also confirmed quatitatively.

We note that in the first dataset, all three dimensions are equally important for the representation of the 8 classes, i.e., the dimensionality can hardly be reduced without losing significant amount of information. In fact, the three dimensions after the KLT contain roughly the same eigenvalues representing percentage of energy contained: 0.37%, 0.33%, 0.30%, i.e., reduction of dimensionality from 3 to 2 will impose a significant challenge for the subsequent classification in the space of reduced dimensionality, which can be measured by the separability and classification error rate, as listed in the table below for both autoencoder and KLT.

We see that for both datasets, the performance of the autoencoder (to the left of the slash symbol) is always slightly better than that of the PCA method (to the right of slash) with higher separability and lower error rate.

Separability Error rate
8 classes in octants 1.52/1.51  0.304/0.320
Iris Dataset  1.12/1.07  0.020/0.027

(16.45)

Example 2: In general, the autoencoder network is trained by all samples in the dataset. However, if all samples in a dataset are labeled to belong to one of a set of classes, the autoencoder can also be trained by these classes. Specifically in this example, the three classes in the iris dataset each represented by the average of all samples in the class are used to train the autoencoder. The figure below shows the 4-D iris dataset mapped to 2-D (top) and 3-D (bottom) spaces by both the autoencoder (left) and KLT (right). It is interesting to observe that in the case of autoencoder (left), the data points representing the three classes are arranged into a line structure in both 2 and 3-D spaces.

The same method is applied to the dataset of 10 classes of handwritten digits from 0 to 9, each represented by 224 training samples in image form of 16 × 16 pixels, treated as a 256 dimensional vector. After the autoencoder network is fully trained based on the means of all 224 samples of each of the 10 classes, the dataset containing 224 × 10 = 2240 data samples in the l-dimensional space spanned by the hidden layer nodes that encode the 256-d samples is then visualized in 2 or 3-D space based the KLT transform. The figure below shows such visualizations of the dataset encoded by l = 3 to l = 18 hidden layer nodes. We note that when the dimensionality l is low, amazingly regular patterns (e.g., square and cubical structures) emerge in the spatial distribution of the data samples. The distribution will then remain mostly the same when the dimensionality l is further increased.

The desired sparsity of the hidden layer can be achieved by using a small number of hidden layer nodes, or letting only a small fraction of the hidden layer nodes to be active (but different nodes are active for different inputs). The second approach can be realized by (a) keeping only a small number of hidden layer nodes with maximum outputs while setting the rest to zero, or (b) forcing the average activation of the hidden nodes to be small.

The last method can be realized by including an extra regularization or penalty term in the objective function to encourange low average hidden node activation. To do so, we replace the second term in Eq. (16.24) by one of the following:

• L1 or L2 regularization

J = ε + λ ∑_{j=1}^{l} |a_{j}|, J = ε + \frac{λ}{2} ∑_{j=1}^{l} a_{j}^{2}  (16.46)

• KL-divergnece

We first define the sparsity parameter ρ for the average actiation of a hidden node, typically a small value close to zero (e.g., ρ = 0.05), and also find for the jth hidden layer node its average activation over all sample N points in the given dataset:

ρ̂_{j} = \frac{1}{N} ∑_{n=1}^{N} ρ_{j}(x_{n}) = \frac{1}{N} ∑_{n=1}^{N} g(a_{j}) = \frac{1}{N} ∑_{n=1}^{N} g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) (j = 1, ⋯, l)

(16.47)

where ρ_{j}(x_{n}) = g(a_{j}) = g (∑_{k} w_{jk}^{h}x_{k}) is the activation level of the jth hidden layer node while responding to input x_{n}. The difference between the desired sparsity in terms of the desired average activation ρ and the actual one ρ̂ can be measured by the KL-divergence (Section 3.2.2):

KL(ρ||ρ̂_{j}) = ρ log (\frac{ρ}{ρ̂_{j}} ) + (1 − ρ) log (\frac{1 − ρ}{1 − ρ̂_{j}} )  (16.48)

where ρ is treated as the probability of the Bernoulli distribution for a binary random variable to be x = 1 and 1 − ρ is the probability for x = 0, and so is ρ̂_{j} for another Bernoulli distribution. By minimizing KL(ρ||ρ̂_{j}) for the difference between the two distributions, the actual activity ρ̂_{j} is forced to be close to the desired ρ. The regularization is the sum of the KLdivergences of all l hidden layer nodes: ∑_{j=1}^{l} KL(ρ||ρ̂_{j}). We note that the KL-divergence is a function of hidden layer weights w_{jk}^{h}, but independent of the output layer weights.

The plot below shows the comparison of the KL-divergence and the quadratic function a(ρ − ρ̂)^{2}. While both functions are minimized to zero when ρ̂ = ρ = 0.3, they behave differently as shown in the range of [0, 1].

Similar to Eq. (16.24) for back propagation, we construct the objective function for the autoencoder as the sum of the error ε and the regularization term above to get

J(W^{h}, W^{o}) = ε + λ ∑_{j=1}^{l} KL(ρ||ρ̂_{j})  (16.49)

During the iterative learning the weights are updated based on gradient descent method, same as in back propagation. Given a specific input x we first get the output x̂ through the forward pass, and then modify the weights for both the output and hidden layers in the backward pass based on the gradient of the objective function with respect to the weights. While the computation for updating the weights for the output layer is the same as steps 4 and 5 in the algorithm listed before for back propagation, the computation for the weights for the hidden layer is different from steps 6 and 7 due to the regularization term, as shown below:

\frac{∂}{∂w_{jk}^{h}} J(W^{h}, W^{o}) = \frac{∂ε}{∂w_{jk}^{h}} + λ\frac{∂}{∂w_{jk}^{h}} [∑_{j=1}^{l} KL(ρ||ρ̂_{j})]  (16.50)

The first term above is the same as as that in Eq. (16.28) (excluding the second regularization term), the second regularization term can be found by chain rule as:

\frac{∂}{∂ρ̂_{j}} [∑_{j=1}^{l} ρ log (\frac{ρ}{ρ̂_{j}} ) + (1 − ρ) log (\frac{1 − ρ}{1 − ρ̂_{j}} )] \frac{∂ρ̂_{j}}{∂w_{jk}^{h}}

= [ρ\frac{∂}{∂ρ̂_{j}} (log ρ − log ρ̂_{j}) + (1 − ρ)\frac{∂}{∂ρ̂_{j}} [(log(1 − ρ) − log(1 − ρ̂_{j}))]] \frac{∂}{∂w_{ij}} g (∑_{k=0}^{d} w_{jk}^{h}z_{i})
= (−\frac{ρ}{ρ̂_{j}} + \frac{1 − ρ}{1 − ρ̂_{j}} ) g′(a_{j})x_{k}  (16.51)

Now d_{j}^{h} in Eq. (16.28) for back propagation (defined in Eq. (16.29) ) can be modified as below for the autoencoder:

d_{j}^{h} = [(∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) + λ (−\frac{ρ}{ρ̂_{j}} + \frac{1 − ρ}{1 − ρ̂_{j}} )] g′(a_{j}^{h})  (16.52)

Using this modified d_{j}^{h} in step 6 of the algorithm for the propagation listed in the previous section, we get the algorithms for the autoencoder.

Autoencoder and PCA are similar for XOR and cocentric datasets. but different for data of digits from 0 to 9.

16.4 Deep Learning

Deep learning, a method based on multi-layer feed forward network with multiple hidden layers, has gained popularity and caused a new wave of enthusiasm for artificial neural networks in the recent years, due heavily to the progress in computational capability as well as the latest development in new algorithms.

Conceptually, deep learning network can be simply understood as a network composed of multiple learning layers, including more than one hidden layer as well as the output layer. The perceptron network with a single learning layer and the three-layer network with the hidden and output learning layers discussed previously can be considered as special cases of deep learning. If we model a learning layer by a nonlinear function that maps an input x to an output

y = g(w^{T} x) = fx, w)  (16.53)

than an L-layer deep learning network can simply be described by

y = f^{(L)}(f^{(L−1)}(⋯ f^{(1)}(x, w^{(1)}), w^{(L−1)}), w^{(L)})  (16.54)

all for the purpose of modeling some desired relationship between input x and output y. When the multiple layers in a deep learning network are fully connected (FC) , i.e., a node in a layer gets inputs from all nodes in the previous layer weighted by w, then the method of back propagation discussed previously is used to update the weights based on gradient descent.

The basic structure of the multi-layer network has been inspired by the biology of the brain. The hierarchical structure of the network is composed of lower level layers where the neurons selectively respond to some simple local features in the data, while neurons in subsequent higher lavel layers selectively respond to progressively more complex and global patterns. Such a hierarchy is very similar to the two main visual pathways in the primate brain, known as the dorsal or “where” pathway and the ventral or “what” pathway, both formed by multiple levels of cortical areas starting from the primary visual cortex (V1), each composed of different types of neurons in terms of the size and specific type of the visual stimuli they are selectively responsive to.

Specifically, the hierarchy of ventral pathway known for visual object recognition is composed of neurons in the lower layer such as V1 where the simple cells with small receptive field selectively respond to edges of different orientations, and neurons with progressively larger receptive fields in the higher layers selectively respond to more complex features in larger areas of the visual field, such as neurons in the inferior temporal (IT) area sensitive to visual patterns such as image of faces.

As an example of deep learning networks, the cvolutional neural network (CNN) is composed of multiple 2-D layers for the purpose of image object recognition. This network can be considered to a large extent a mimic of the ventral pathway of the visual cortex described above, in the sense that the nodes in the lower layers extract local features in the data (mimicing the simple cells in V1), while the nodes in higher layers respond selectively to more global patterns represented in terms of the local features detected by the lower layers (mimicing the cells in IT).

Specifically, in the lower layers of the CNN, local image features such as edgesare detected by convolution, a basic technique commonly used in image processing. Mathematically, convolution is an operation defined for two functions f(t) and g:

y(t) = x(t) ∗ h(t) = ∫_{−∞}^{∞} x(t − τ)h(τ)dτ = ∫_{−∞}^{∞} h(t − τ)x(τ)dτ  (16.55)

which is generalized to 2-D discrete functions in image processing:

y[m, n] = ∑_{i=−k}^{k} ∑_{j=−k}^{k} x[m − i, n − j] h[i, j]  (16.56)

where h[−i, −j] = h[i, j], called convolution kernel or sometimes filters, is a central symmetric function defined over −k ≤ i, j ≤ k, represented by a (2k + 1)×(2k +1) square matrix, and y[m, n] is a pixel of the image y = h∗x obtained by convolving image x by kernel h. The figure below illustrates 2-D convolution with a 3 × 3 kernel.

We see that each pixel y[m, n] of the resulting image y is the sum of 9 products. As the kernel slides over the input image x pixel by pixel and then row by row, all pixels of y are calculated.

These four kernels are for detecting edges in vertical and horizontal orientations:

[ \begin{matrix}−1 & 0 & 1 \\ −1 & 0 & 1 \\ −1 & 0 & 1 \end{matrix} ] , [ \begin{matrix}1 & 1 & 1 \\ 0 & 0 & 0 \\ −1 & −1 & −1 \end{matrix} ] , [ \begin{matrix}1 & 0 & −1 \\ 1 & 0 & −1 \\ 1 & 0 & −1 \end{matrix} ] , [ \begin{matrix}−1 & −1 & −1 \\ 0 & 0 & 0 \\ 1 & 1 & 1 \end{matrix} ]

(16.57)

while these four kernels are for detecting edges in two diagonal orientations:

[ \begin{matrix}0 & 1 & 1 \\ −1 & 0 & 1 \\ −1 & −1 & 0 \end{matrix} ] , [ \begin{matrix}1 & 1 & 0 \\ 1 & 0 & −1 \\ 0 & −1 & −1 \end{matrix} ] [ \begin{matrix}0 & −1 & −1 \\ 1 & 0 & −1 \\ 1 & 1 & 0 \end{matrix} ] , [ \begin{matrix}−1 & −1 & 0 \\ −1 & 0 & 1 \\ 0 & 1 & 1 \end{matrix} ]

(16.58)

The figure below shows the effects of edge detection by convolution. The cat image is convolved with the kernels for in Eq. (16.57), while the panda image is convolved with kernels in Eq. (16.58).

We note that the convolution as a linear combination is in the same general operation y = f(x, w) in neural networks with the elements in the kernel treated as the weights, by which x as the output from a lower arranged as a 2-D array is fed forward into the next layer that produces its output y.

These 3 × 3 kernels can be extended to larger kernels for detecting more complex features in a larger area of the visual field in higher layers of the network.

imperical operations:

• Convolution:

Each node in the convolution layers is only locally connected to a set of neighboring nodes in a square patch in the previous layer. In particular, a node in the first layer take as input the pixel values inside its receptive fields, a subregion of the image. It then generates the convolution, the sum of the inputs weighted by the kernel components, and feed it to the next layer. At each position, multiple nodes with the same receptive field form a column in the depth dimension, all performing convolution but based on different kernel to extract different types of local features, such as edges in various orientations as shown above.

• Padding:

Padding is needed for the output of convolution to maintain the same size as the input. For example, for a 5 × 5 kernel, the first pixel at the top-left corner of the output is in the position of the pixel in the 3th row and 3th column in the input, i.e., the output is 4 pixels smaller in both dimension than the input. To prevent this size reduction from one layer to the next, the input needs to be padded with additional rows and columns of pixels around its four borders (zeros or taking same value as the border pixels).

• Stride:

Stride s is the number of pixels for the revolution kernel to shift in both dimensions in convolution from one . In regular convolution, s = 1, the size of the resulting image is the same as the input. But when s > 1, the kernel shifts s pixels from the center of one receptive field to next. In other words, stride is the distance between the receptive field centers of neighboring neurons. The size of the output will be 1/s of the original in both dimensions.

• Pooling:

Pooling is a down-sampling method by which a local patch (e.g., 2 × 2) in the previous layer is represented by their average or maximum as a single value in the next layer called pooling layer, for dimension reduction that serves two purposes: (a) local shift and rotational invariance and (b) computation reduction.

• Dilution or Dropout:

Certain fraction of nodes randomly selected in a layer is dropped by setting the corresponding weights to zero with a set probability, for the purpose of preventing overfitting, so that the network is less sensitive to certain detailed variations in the data due most likely to noise and thereby more generalizable.

• In the last stage of the CNN, all nodes are fully connected and they taking progressively larger and more complex features extracted by the lower layers as inputs, these fully connected layers to carry out the final recognition based on backpropagation.

• Weight decay:

As mentioned before, a multilayer neural network can be considered as a function y = f(x, w) for modeling the relationship between x and y, same as many supervised learning algorithms regression and classification. Similarly, including a penalty term proportional to ||w|| or its square in the objective function is an effective way to regulate the algorithm and avoid overfitting.

16.5 Generative Adversarial Networks

https://arxiv.org/pdf/1701.00160.pdf https://developers.google.com/machine-learning/gan https://www.youtube.com/watch?v=vi7lACKOUao http://neuralnetworksanddeeplearning.com/

In general, generative methods try to learn the the probability density function (PDF) as a model for the data of interest. But in the case of generative adversarial network GAN, the network learns to generate samples from the pdf instead of the specific form of the pdf.

The GAN is essentially a two-player minimax game in which two players, the discriminator and the generator, try to achieve their own goals adversarial to each other.

• The discriminator is a binary classifier represented by a function D(x; θ_{D})

that maps an input x to a real value between 1 and 0, indicating respectively whether x is a real smaple from the dataset, or it is a fake sample artificially generated. In principle the discriminator could be any kind of binary classifier with parameters which can be improved by gradient descent during the training process. But it is typically assumed to be deep neural network.

• The generator is modeled as a function x = G(z; θ_{G}) that maps an input z, a random vector (latent variable) with lower dimension then x, into another vector x, a fake sample that mimics the real samples as close to those real one from the dataset as possible so that the discriminator cannot distiguish between the real and the fake. The input z controls various variations in the outputs of the generator. Typically the generator is also a deep neural network that maps random input z ot a desired output.

The goal of the learning process is to reach the Nash equilibrium, a term in game theory, at which neither the discriminator nor the generator can improve its performance any further in terms of what they each try to achieve. Specifically, the discriminator can no longer distinguish between the fake data samples from the generator and the real samples from the dataset, i.e., the model of the given dataset is estimated in terms of the new samples drawn from the learned pdf, although the pdf itself is not explicitly specified. The discriminator is only to help train the generator, once the generator is fully trained, the discriminator is no longer needed.

The discriminaotor tries to maximinize the expectation of its output D(x) when the input is real but minimize the expectation of its output D(Gz), or maximize 1 − D(Gz), when the input is fake from the generator, i.e., the discriminaotor is to maximize the following objective function:

J(D, G) = E_{p(x)}[log D(x, θ_{D})] + E_{p(z)}[log(1 − D(G(z, θ_{G})))]  (16.59)

while the genreator is to minimize it, i.e., the goal of the overall GAN is

min_{G} max_{D} J(D, G)  (16.60)

Convergence of GANs is hard to reach as the discriminator and the generator are each optimizing a different objective function.

used mostly in image data, deep convolutional GAN, to generate large images (e.g., human faces) (e.g., a z-vector z_{man+glasses} − z_{man} + z_{woman} may produce output of woman with classes.)

17 Competition-Based Networks

17.1 Competitive Learning Network

Competitive learning is a neural network algorithm for unsupervised clustering, similar to the K-means algorithm considered previously. The competitive learning takes place in a two-layer network composed of an input layer of d nodes that receives an input vector x = [x_{1}, ⋯, x_{d}]^{T} as a point in the d-dimensional feature space, and an output layer of m nodes that produces an output pattern y = [y_{1}, ⋯, y_{m}]^{T} representing the clusters of interest. Each input variable x_{i} may take either a continuous real value or a binary value depending on the specific application, the outputs y_{i} ∈ {0, 1} are binary, of which only one is 1 while all others are 0 (one-hot), as the result of a winner-take-all completition based on their activation values.

In each iteration of the learning process, a pattern x randomly chosen from the unlabeled dataset X = [x_{1}, ⋯, x_{N}] is presented to the input layer of the network, while each node of the output layer gets an activation value, a linear combination of all such d inputs, i.e., the inner product of its weight vector w_{i} and the input vector x:

a_{i} = ∑_{j=1}^{d} w_{ij}x_{j} = w_{i}^{T} x = ||w_{i}|| ||x|| cos θ (i = 1, ⋯, m)  (17.1)

where θ is the angle between vectors w_{i} and x in the d-dimensinal feature space. If the data vectors are optionally normalized with unit length ||x||^{2} = 1, i.e., they are points on the unit hypersphere in the space. Optionally if the weight vectors w_{i} is also normalized with ||w_{i}|| = 1, then their inner product above is only affected by the angle θ between them.

The output of the network is determined by a winner-take-all competition. The node in the output layer that is maximally activated will output 1 while all others output 0:

y_{i} = \{ \begin{matrix}1 & if a_{i} = w_{i}^{T} x = max_{j} w_{j}^{T} x \\ 0 & otherwise \end{matrix} , (i = 1, ⋯, K)  (17.2)

The inner product w_{i}^{T} x is closely related to the Euclidean distance d(x, w) = ||w_{i} − x||:

d^{2}(w_{i}, x) = ||w_{i} − x||^{2} = (w_{i} − x)^{T} (w_{i} − x) = w_{i}^{T} w − 2w_{i}^{T} x + x^{T} x
= ||w_{i}||^{2} + ||x||^{2} − 2w_{i}^{T} x = ||w_{i}||^{2} + ||x||^{2} − 2||w_{i}|| ||x|| cos θ (17.3)

If both x and w_{i} are normalized, the winning node with the maximum activation a_{i} = w_{i}^{T} x also has the minimum Euclidean distance ||w_{i} − x|| and anglular distance θ. Therefore the competition above can also be expressed as the following:

y_{i} = \{ \begin{matrix}1 & if ||w_{i} − x|| = min_{j} ||w_{j} − x|| \\ 0 & otherwise \end{matrix} , (i = 1, ⋯, K)  (17.4)

The only winning node that receives the highest activation gets to modify its weight vector:

w_{i}^{new} = w_{i}^{old} + y_{i} η (x − w_{i}^{old}) = \{ \begin{matrix}(1 − η)w_{i}^{old} + ηx & if y_{i} = 1 \\ w_{i}^{old} & if y_{i} = 0 \end{matrix}  (17.5)

where 0 < η < 1 is the learning rate (step size). Optionally the new weight vector can also be renormalized. For the iteration process to gradually approach a stable clustering result, the learning rate η needs to be reduced through the iterations from its initial value (e.g., η_{0} = 0.9) toward 0 by a certain decay factor α (e.g., α = 0.99). This way the learning rate decays exponentially for the learning process to gradually slow down and becomes stabalized. The decay factor α depends on the size of the dataset, and it may need to be determined heuristically and experimentally.

The competitive learing is illustrated in the figure above, where the modified weight vector of the winner w_{i}^{new} is between x and the old w_{j}^{old}, i.e., the effect of the learning process is that the winner’s weight vector which is closest to the current input pattern x is pulled even closer to it.

Here are the iterative steps in the competitive learning:

• Step 0: Normalize all data vectors x, initialize randomly the weight vectors w for the output nodes (e.g., any m randomly chosen samples from the dataset): w_{1}^{(0)}, w_{2}^{(0)}, ⋯, w_{m}^{(0)}, set iteration index to zero l = 1;

• Step 1: Choose randomly an input pattern x from the dataset and calculate the activation for each of the output nodes:

a_{i} = w_{i}^{T} x (i = 1, ⋯, m)  (17.6)

• Step 2: find the index of the winning node j = arg max_{i} a_{i}, and update its

weights:

w_{j}^{(l+1)} = w_{j}^{(l)} + η(x − w_{i}^{(l)})  (17.7)

Reduce learning rate η ← αη. Optionally, renormalize w_{j}^{(l+1)}.

• Step 3: Terminate the iteration if the clustering result has gradually stablized, when the learning rate η is reduced from its initial value η_{0} to some small value (e.g., 0.1) and the weight vectors no longer change significantly. Otherwise l ← l + 1, go back to Step 1.

This iterative process can be more intuitively understood as shown in the figure below. Every time a sample x (a red dot) is presented to the input layer, one of the output nodes will become the winner and its weight vector w (an blue x) closest to the current input x is drawn even closer to x. As this process is carried out iteratively many times, each cluster of similar sample points in the space will draw one of the weight vectors towards its centeral area, and the corresponding output node will always win and output 1 whenever any member of the cluster is presented to the network in the future. In other words, after this unsupervised learning, the feature space is partitioned into K regions each corresponding to one of the K clusters, represented by one of the output nodes, whose weight vector is in the central area of the region. We see that this process is very similar to the K-means clustering algorithm, in which each cluster is represented by the mean vector of all of its members.

It is possible in some cases that the data samples are not distributed in such a way that they form a set of clearly saperable clusters the feature space. In the extreme case, they may even form a continuum. In such cases, a small number of output nodes (possibly even just one) may become frequent winners, while others become “dead nodes” if they never win and consequently never get the chance to odify their weight vectors. To avoid such meaningless outcome, a mechanism is needed to ensure that all nodes have some chance to win. This can be implemented by including an extra bias term in the learning rule:

a_{i} = w_{i}^{T} x + b_{i}  (17.8)

where b_{i} is the bias term proportional to the difference between the “fair share” of winning 1/m and the actual winning frequency:

b_{i} = c (\frac{1}{m} − \frac{number of winnings of the ith node}{total number iterations so far})  (17.9)

The value of the bias term b_{i} for the ith node will change its winning frequency. If the node is winning more than its share 1/m, then b_{i} < 0 and it becomes harder for it to win in the future. On the other hand, if the node rarely wins, b_{i} > 0 and its chance to win in the future is increased. Here hyperparameter c is some scaling coefficient. The greater c, the more balanced the competition will be. It needs to be fine tuned based on the specific nature of the data being analyzed.

This process of competitive learning can also be viewed as a vector quantization process, by which the continuous vector space is quantized to become a set of K discrete regions, called Voronoi diagram (tessellation). Vector quantization can be used for data compression. A cluster of similar signals x in the vector space can all be approximately represented by the weight vector w of one of a small number of K output nodes in the neighborhood of x, thereby the data size can be significantly reduced.

The Matlab code for the iteration loop of the algorithm is listed below.

    [d N]=size(X);                 % dataset containing N samples
    b=zeros(1,K);                  % bias terms for K clusters
    freq=zeros(1,K);               % winning frequencies
    eta=0.9;                       % initial learning rate
    decay=0.99;                    % decay rate
    it=0;
    while eta>0.1                  % main iterations
        it=it+1;
        W0=W;                      % initial weight vectors
        x=X(:,randi(N,1));         % randomly select an input sample
        dmin=inf;
        for k=1:K                  % find winner among all K output nodes
            d=norm(x-W(:,k))-b(k);
            if dmin>d
                dmin=d; m=k;       % mth node is the winner
            end
        end
        w=W(:,m)+eta*(x-W(:,m));   % modify winner's weights
        W(:,m)=w/norm(w);          % renormalize its weights (optional)
        share(m)=share(m)+1;       % update share for winner
        b=c*(1/K-share/it);        % modify biases for all nodes
        eta=eta*decay;             % reduce learning rate
    end

Examples

The competitive learning method is applied to a set of simulated 2-D data of four clusters. The network has d = 2 input nodes and K = 4 output nodes. The first 16 iterations are shown in the figure below, where open circles represent the data points, while the solid back squares represent the weight vectors. Also, the blud and red squares represent the weight vector of the winner before and after its modification, visualized by the straight line connecting the two squares. We see that the K = 4 weight vectors randomly initialized are iteratively modified one at a time, and after these 16 iterations, they each move to the center of one of the four clusters. The saperability of the clustering result measured by trS_{T}^{−1}S_{B} is 1.0. When the number of output nodes is reduced to 3 and 2, the saperability is also reduced to 0.76 and 0.46, respectively. When K = 5 output nodes are used, one of the four clusters is represented by two output nodes.

The same method is applied to a set of 3-D data of eight clusters. The results are shown in the figure below, where the top and bottom rows show the data and weight vectors before and after clustering, respectively. The left column shows the 2-D space spanned by the first two principal components, while the right column shows the original 2-D space. Again we see that the weight vectors move from their random initial locations to the centers of the eight clusters as the result of the clustering.

The competitive learning algorithm is also applied to the dataset of handwritten digits from 0 to 9, and the clustering result is shown as the average of all samples in each cluster in 16 × 16 image form shown in the figure below.

17.2 Self-Organizing Map (SOM)

The Self-organizing map (SOM) is a two-layer unsupervised neural network learning algorithm that maps any input pattern presented to its input layer, a vector x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional feature space, to a set of output nodes that forms a low-dimensional space called feature map, typically a 2-D grid (lattice), although 1-D and 3-D spaces can also be used. A SOM can therefore be considered as a dimensionality reduction method, by which the dataset in the high-dimensional feature space can be visualized and the features and structures of the dataset can be discovered.

The learning rule of the SOM is similar to that of the competitive learning network considered previously in that all output nodes compete based on their activation levels upon the input pattern presented to input nodes of the network. However, different from the competitive learning network where only the winning node gets to modify its weight vector and learn to respond to a cluster of similar patterns individually and independently, here in SOM network all output nodes in the neighborhood of the winning node get to modify their weight vectors and thereby learn to respond collectively to a cluster of similar input patterns. As the result, all output nodes are locked into a continum feature map with smooth and gradual variation in responsive selectivity.

The SOM as a learning algorithm is motivated by the mapping process in the brain, by which signals from various sensory systems (e.g., visual and auditory) are projected (mapped) to different cortical areas and responded to by the neurons wherein in such a way that all neurons in a local cortical area selectively respond to similar stimuli, as shown in the following examples.

• Spatially adjacent stimuli are represented and responded to by neurons in adjacent positions in cortex, known as topographic organization.

• Sound signals of different frequencies are mapped to the primary auditory cortex where neighboring neurons respond to similar frequencies.

• The visual field is topographically projected from the retina to the primary visual cortex V1 where adjacent neurons respond to adjacent areas in the visual field. This projection of the visual field is called retinotopic map.

• Visual edges/lines of different orientations are mapped to V1 where neighoring neurons in the columnar architecture respond to similar orientations.

• Visual motion of different directions are mapped to the columnar architecture of the middle temporal area (MT or V5) where neighboring neurons respond to similar motion directions.

In a typical SOM network, the output nodes are organized in a 2-D feature map, and the competitive learning algorithm previouisly discussed is modified so that the learning takes place at not only the winning node in the output layer, but also a set of nodes in the neighborhood of the winner. The weight vectors of all such nodes are modified:

w_{i}^{new} = w_{i}^{old} + u_{i}η (x − w_{i}^{old})  (17.10)

where u_{i} is a weighting function based on the Euclidean distance d_{i} from the ith node in the neighborhood to the winner in the 2-D feature map, such as a Gaussian function centered at the winner:

u_{i} = exp (−\frac{d_{i}^{2}}{2σ^{2}} ) \{ \begin{matrix}= 1 & for winner with d_{i} = 0 \\ < 1 & for all orther nodes \end{matrix}  (17.11)

where σ is a hyperparameter that controls the width of the Gaussian function. We see that all neurons in the neighborhood of the winner learn from the current input pattern in such a way that their weight vectors are modified to be closer to the input vector in the d-dimensional feature space, but they do so to different extents. Speically, for the winner itself with d_{i} = 0 and u_{k} = 1, the learning rate for the winner is maximally η, while all other nodes with d_{i} > 0 also modify their weights but with some lower rates u_{i}η. Those closer to the winner with smaller d_{i} and thereby greater u_{i} will learn more than those farther away.

Here are the specific steps of the SOM learning algorithm:

1. Initialize weights of a set of output nodes arranged in a 2-D map.

2. Choose randomly an input pattern vector x in the high-dimensional feature space and calculate the activation y_{i} = w_{i}^{T} x for each of the output nodes.

3. Find the winning node with maximum activation y_{k} and update the weights of all nodes in its neighborhood by Eq. (17.10).

4. Go back to step 2 until the feature map is no longer changing or a set maximum number of iterations is reached.

In the training process, both σ controlling the size of the neighborhood and η for the learning rate will be gradually reduced.

Shown below is the the Matlab code segment for the essential part of the SOM algorithm.

    [d N]=size(X);                         % dataset of N sample vectors
    W=rand(d,M,M);                         % initialization of weights
    for i=1:M
        for j=1:M
            w=reshape(W(:,i,j),[d 1]);
            W(i,j,:)=W(:,i,j)/norm(w);     % normalize all weight vectors

end
end
eta=0.9;                               % initial learning rate
sgm=M;                                 % width of Gaussian neighborhood
decay=0.999;                           % rate of decay
for it=1:nt                            % training iterations
n=randi([1 N]);
x=X(:,n);                          % pick randomly an input
amax=-inf;
for i=1:M                          % find the winner
    for j=1:M
        w=reshape(W(:,i,j),[d 1]);
        a=x'*w;                    % activation as inner product
        if a>amax
            amax=a;  wi=i;  wj=j;  % record winner so far
        end
    end
end
for i=1:M
    for j=1:M
        dist=(wi-i)^2+(wj-j)^2;    % distance to winner
        c=exp(-dist/sgm);          % Gaussian weights
        w=reshape(W(:,i,j),[d 1]); % get a weight vector
        w=w+eta*c*(x-w);           % modify weights
        w=w/norm(w);               % normalize weight vectors
        W(:,i,j)=w;                % save modified weight vector
    end
end
eta=eta*decay;                     % reduce learning rate
sgm=sgm*decay;                     % reduce width of Gaussian
end

Example 1: This example of “self-organizing map” is how the SOM algorithm got its name.

The output nodes are organized into a 2-D map and they learn to respond to the positions of a set of random points x = [x_{1}, x_{2}]^{T} in a 2-D feature space. The weights of the output nodes are randomly initialized, and then modified during the learning process when the data points are repeatedly presented to the input of the network. When the SOM hase been trained, due to the spatial correlation nature of the algorithm, the output nodes close to each other in the 2-D feature map respond to a set of adjacent data points in the 2-D space, thereby forming a self-organized spatial map, a roughly regular grid.

The 2-D map of the output nodes is visualized as shown in the figure below, where the spatial position of each node is determined by the two components of its weight vector w = [w_{1}, w_{2}]^{T} treated as its coordinates, and each node is connected to its four neighbors in the 2-D grid to indicate the spatial structure of the feature map.

In this example, the competitive learning is based on the distance between the weight vector w_{i} and the current input x, instead of their inner product w_{i}^{T} x. The node with the shortest distance ||w_{i} − x|| becomes the winner, and its weight vector w is pulled even closer to the input, dragging along with it all its neighboring nodes.

The figure below shows the iterative modification of the weights. In each of the 16 iterations, the output node (blue dot) is closest to the input vector (red dot) and it becomes the winner.

The figure below shows the subsequent iterations in every 50 iterations. The resulting configuration of the output nodes is roughly a regular 2-D grid, a selforganized map in the 2-D feature space.

Example 2:

In this example the output nodes of the SOM are arranged into a 1-D loop and they respond to a set of cities in north America represented by their coordinates (longitudes and latitudes), for the purpose of finding a reasonably short path that goes through all of these cities (the traveling salesman problem). These nodes are visualized in terms of their weight vectors as points in a 2-D map. The weights are initialized to form circle in the map the cities. During the learning iteration when they learn to respond to the 2-D city coordinates as the input, their weights are modified so that they are gradually pulled toward the cities, thereby forming eventually a path through all the cities. In this case, the inner product w_{i}^{T} x is used to determine the winner of each iteration.

The figure below shows the results of the first 16 iterations:

The figure below shows the subsequent iterations in every 50 iterations. The resulting configuration of the output nodes approaches the desired path that passes through all cities. Although this result does not garantee the path to be the shortest, it is reasonably close to such a solution required by the traveling salesman problem. The final result is unaffected by how the weights are initalized.

Example 3:

In this example, the output nodes in the 2-D feature map learn to respond to different colors treated as vectors in a 3-D space spanned by the three primary colors red, green and blue (RGB). When the SOM is trained, neighboring output nodes respond to similar colors. Here all input vectors are normalized (the hue of a color is fully determined by the ratios of the three primaries), and so are the weight vectors, which are always renormalized in each iteration.

To visualize the output nodes in the 2-D map, they are color coded based on the three components of their weight vectors treated as the three RGB colors, indicating their favored colors, the color they respond maximally, as shown in the figure below. The left panel is the nodes in the feature map with randomly initialized weights. The middle one shows these nodes after the SOM is fully trained. The right panel shows all the winners during the learning process, all encoded by the input colors they win. The black areas are composed of the output nodes that never win but learn along with their neighboring winners. Interestingly, the winners seem to form some structure in the 2-D space. More detailed discussion can

be found here: http://fourier.eng.hmc.edu/e161/lectures/NeuralNetworks/node15.html.
  More examples of SOM can be found here: https://en.wikipedia.org/wiki/Self-organizing_map.

Part VI Reinforcement Learning 18 Introduction to Reinforcement

Learning

18.1 Reinforcement Learning

Reinforcement learning (RL) can be considered as one of the three basic machine learning paradigms, alongside supervised learning (e.g., regression and classification) and unsupervised learning (e.g., clustering) discussed previously. The goal of RL is for the algorithm, a software agent, to learn to make a sequence of decisions, called policy, for a specific task in a given environment, for the purpose of receiving the maximum rewards from the environment.

Different from supervised learning for either regression or classification based on a given training dataset composed of a set of i.i.d. sample points x_{n} labeled by y_{n}, (n = 1, ⋯, N), RL is a learning method without any examples of optimal behaviors. Instead, RL learns on its own without relying on any labeled training dataset explicitely informing the agent what the correct responses are while interacting with the environment. On the other hand, different from the unsupervised methods that learns passively from the given dataset, RL is a trialand-error learning method, in which the agent actively interacts with and learns from the given environment.

Specifically, RL as a sequential method depends on the dynamics of the environment which is modeled by a Markov decision process (MDP), a stochatic system of multiple states with probabilistic state transitions and rewards. If the MDP model of the environment in terms of its the state transition and reward probabilities is known, the agaent only needs to find the optimal policy in terms of what action to take at each state of the MDP to achieve maximum accumulated rewards as the consequence of its actions. The task in this model-based case is called planning. On the other hand, if MDP of the environment is unknown, the agent needs to learn the environment by repeatedly running the dynamic process of the MDP based on some initial policy and gradually evaluate the received rewards and improve the policy to eventually reach optimality. The task in this model-free case is called control.

Depending on the action taken by the agent in the current state s, the system transits to the next state s′ following certain transition probability distribution. The RL is to choose the optial action in the current state, to maximize the long-term expected reward, as the feedback of the environment. This is typically carried out by dynamic programming (DP), a class of algorithms seeking to simplify a complex problem by breaking it up into a set of sub-problems which can be solved recursively.

The figure below illustrates the basic concepts of RL, where the agent makes a decision in terms of the action a_{t} to take in the current state s_{t} by following some policy π(a|s), while the environment makes a transition from the current state s_{t} to the next state s_{t+1} by following the transition probability, a conditional probability p(s′|s, a) for the next state s_{t+1} = s′ given the curent state s_{t} = s and the action a_{t} taken by the agent, and provides a reward r_{t+1}.

18.2 Markov Decision Process

Markov decision process (MDP) is the mathematical framework for reinforcement learning. To understand MDP, we first consider the basice concept of Markov process or Markov chain, a stochastic model of a system that can be characterized by a set of states S = {s_{1}, ⋯, s_{N}} of size |S| = N (cardinality of S). The dynamics of the system in terms of its state transition is modeled by the transition probability P_{ij} = P (s_{j}|s_{i}) from the current state s_{t} = s_{i} to the next state s_{t+1} = s_{j} with i, j = 1, ⋯, N. The system is assumed to be memoryless, i.e., its future is independent of the past history h_{t} = {s_{t}, s_{t−1}, ⋯, s_{0}}, given the present s_{t}:

P (s_{t+1}|h_{t}) = P (s_{t+1}|s_{t}, s_{t−1}, ⋯, s_{0})  (18.1)

For example, if the state of a helicopter is described by its linear and angular position and velocity based on which its future position and velocity can be completely determined, then it can be modeled by a Markov process; but if the state is only described by its position, then its position in the past is needed for determining its future (e.g., to find its velocity), and it is not a Markov process.

All transition probabilities can be organized as an N × N state transition matrix:

P = [ \begin{matrix}P_{11} & ⋯ & P_{1N} \\ ⋮ & ... & ⋮ \\ P_{N1} & ⋯ & P_{NN} \end{matrix} ]  (18.2)

As the transition probabilities from any current state s_{i} to the N possible next states s_{1}, ⋯, s_{N} have to sum up to 1, we have

∑_{j=1}^{N} P_{ij} = ∑_{j=1}^{N} P (s_{j}|s_{i}) = 1,  (i = 1, ⋯, N)  (18.3)

Any matrix satisfying this property is called a stochastic matrix.

In the following, our discussion is mostly concentrated on the the general transition from the current state s_{t} = s to the next state s_{t+1} = s′ with transition probability P (s_{t+1} = s′|s_{t} = s) = P_{ss′}.

Example Weather in Los Angeles:

P = [ \begin{matrix}0.7 & 0.2 & 0.1 \\ 0.4 & 0.3 & 0.3 \\ 0.5 & 0.2 & 0.3 \end{matrix} ]  (18.4)

We next consider a Markov reward process (MRP), represented by a tuple ⟨S, P, R, γ⟩ of four elements for the states, state transition probabilities, rewards, and the discount factor. Here the reward R can be considered as a feedback signal to the agent at each time step of the sequence of state transitions, indicating how well it is doing with respect to the overall task. The reward can be negative, as a penalty, for situations to be avoided. The performance of the agent is measured by the sum of rewards accumulated over all time steps of the Markov process, weighted by the discount factor γ ∈ [0, 1] that discounts future rewards. If γ is close to 0, then the immediate reward is emphasized (short-sighted or greedy), but if γ is close to 1, then rewards in the future steps will be almost as valuable as immediate ones (far-sighted).

The sequence of state transitions of an MRP from an initial state s_{0} to a terminal state s_{T} is called an episode, and the number of time steps T is called horizon, which can be either finite or infinite.

Both the reward r and the next state s′ resulting from arriving at the current state s are considered as random variables with joint probability p(s′, r|s) = P (s_{t+1} = s′, r_{t+1} = r|s_{t} = s), based on which both the state transition probability and state reward probability can be found by marginalization:

p(s′|s) = P (s_{t+1} = s′|s_{t} = s) = ∑_{r∈R} p(s′, r|s), P (r|s) = ∑_{s′∈S} p(s′, r|s)

(18.5)

Note that conventionally the reward r received after arriving at state s_{t} is denoted by r_{t+1}, instead of r_{t}. The index s′ ∈ S for the summation over all states will be abbreviated to s′ in the subsequent discussion. The expectation of the reward at s is

r(s) = E[r_{r+1}|s_{t} = s] = ∑_{r} rP (r|s) = ∑_{r} r ∑_{s′} P (s′, r|s)  (18.6)

We define the return G_{t} at each step s = s_{t} as the accumlate reward, the sum of the immediate reward after arriving at the current state s = s_{t} and all delayed rewards in the future states upto a terminal state s_{T} with reward r_{T} at the end of the episode, discounted by γ:

G_{t} = r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ + γ^{T −t−1}r_{T} = ∑_{k=0}^{T −t−1} γ^{k}r_{t+k+1}  (18.7)

We further define the state value function v(s) at state s = s_{t} as the expectation of the return, which can be further expressed recursively in terms of the values v(s′) of all possible next states s′ = s_{t+1}:

v(s) = E[ G_{t}|s_{t} = s ] = E[r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ |s_{t} = s]
= E[r_{t+1} + γ(r_{t+2} + γr_{t+3} + ⋯)|s_{t} = s]
= E[r_{t+1}|s = s_{t}] + γE[G_{t+1}|s_{t+1} = s′, s_{t} = s]
= r(s) + γ ∑_{s′} P (s′|s)v(s′)  (18.8)

where r(s) is given in Eq. (18.6). The value of a terminal state s_{T} at the end of an episode is zero, as there will be no next state and thereby no more future reward.

This equation is called the Bellman equation, by which the value v(s) at current state s is expressed recursively in terms of the immediate reward r(s) and the values v(s′) of all possible next states, without explicitely invoking all future rewards. Based on this bootstrap idea of the Bellman equation, a multi-step MRP problem can be expressed recursively as a subproblem concerning only a single state transition from s to s′, as a backward induction to find current state value from all future ones. For this reason, the Bellman equation plays an essential role in future discussion of some important algorithms in reinforcement learning.

The Bellman equation in Eq. (18.8) holds for all states s_{1}, ⋯, s_{N}, and the resulting N equations can be expressed in vector form as:

v = [ v(s_{1}) ; ⋮ ; v(s_{N}) ] = [ r(s_{1}) ; ⋮ ; r(s_{N}) ] + γ [ \begin{matrix}P (s_{1}|s_{1}) & ⋯ & P (s_{N}|s_{1}) \\ ⋮ & ... & ⋮ \\ P (s_{1}|s_{N}) & ⋯ & P (s_{N}|s_{N}) \end{matrix} ] [ v(s_{1}) ; ⋮ ; v(s_{N}) ]
= r + γPv  (18.9)

where P is called the stochastic matrix. Solving this linear equation system of size N we find

v = (I − γP)^{−1}r  (18.10)

Such a solution exists as matrix I − γP is invertible. This can be shown by noting that all eigenvalues of the stochastic matrix P are not greater than 1: |λ| ≤ 1 (Section A.9), and all eigenvalues of γP are smaller than 1: γλ < 1, i.e., the eigenvalue of I − γP is greater than zero: 1 − γλ > 0. Consequently the determinant det(I − γP) of this coefficient matrix, as the product of all its nonzero eigenvalues, is non-zero and therefore matrix det(I − γP) is invertible.

Alternatively, the Bellman equation in Eq. (18.8) can also be solved iteratively by the general method of dynamic programming (DP), which solves a multistage planning problem by a backward induction and find the value function recursively. We first rewrite the Bellman equation as

v = r + γPv = B(v)  (18.11)

where B(v) = r + γPv is defined as a vector-valued function (an operator) applied to the vector argument v. Now the Bellman equation can be solved iteratively from an arbitrary initial value, such as v_{0} = 0:

v_{n+1} = B(v_{n}) = r + γPv_{n}  (18.12)

This iteration will always converge to the root of the equation, the fixed point of function B(v), as it is a contraction mapping first introduced in Section 1.3, that satisfies:

||B(v_{i}) − B(v_{j})|| = ||r + γPv_{i} − (r + γPv_{j}) || = γ||P(v_{i} − v_{j})||
≤ γ||P|| ||v_{i} − v_{j}|| = γ||v_{i} − v_{j}|| < ||v_{i} − v_{j}|| (18.13)

That B(v) is a contraction can also be proven by showing the norm of its Jacobian matrix, its derivative with respect to its vector argument v, is smaller than 1. We first find the Jacobian matrix

J_{B} = \frac{d}{dv} B(v) = \frac{d}{dv} (r + γBv) = γP  (18.14)

and then find its p-norm with p = ∞ (equivalent to p = 1, 2), the maximum absolute row sum:

||J_{B}||_{p=∞} = ||γP||_{∞} = γ||P||_{∞} = γ max_{1≤i≤N} ∑_{j=1}^{N} |P_{ij}| = γ < 1  (18.15)

The last equality is due to Eq. (18.3), i.e., ||P||_{∞} = 1.

In summary, the Bellman equation in Eq. (18.8) can be solved by either of the two methods in Eqs. (18.10) and (18.12), so long as γ < 1. When the size of state space |S| = N is large, the complexity O(N^{3}) is too high then the iterative method may be more efficient.

Based on the definition of MRP, we further define Markov decision process (MDP) as an MRP with certain decision-making rule called policy, denoted by π, following the policy the agent takes one of the actions a ∈ A(s) available at each state s to control the state transition of the process. An MDP can therefore be described by a tuple < S, P, R, A, γ >, of which A is the set of all actions. The policy π can be either stochastic denoted by π(a|s) = P (a_{t} = a|s_{t} = s) as the conditional probability of taking action a in state s, or deterministic denoted by a = π(s). A policy is soft if any of the actions available at a state is possible to be taken, i.e., π(a|s) > 0 for all a ∈ A(s) and all s ∈ S. In particular, a policy is ϵ-soft if π(a|s) ≥ ϵ/|A(s)| for some small value of ϵ.

As there are |A(s_{n})| possible actions to take in state s_{n} ∈ S, the total number of policies is |A(s_{1})| ⋯ |A(s_{N})| = Π_{n=1}^{N}|A(s_{n})|. If the number of available actions |A| is the same for all |S| states, then the total number of policies is |A|^{|S|}.

Following a given policy from an initial state s_{0} will result a sequence of state transitions called a trajectory of the episode:

s_{0} ⟶^{a_{0}} s_{1} ⟶^{a_{1}} s_{2} ⟶^{a_{2}} s_{3} ⋯ s_{T}  (18.16)

of which the state visited at the t-th step denoted by s_{t} can be any of the N states in S = {s_{1}, ⋯, s_{N}} of the MDP, not to be confused with the t-th state in S. Note that in an episode some of the states in S may be visited multiple times, while some others may never be visited. Also note that due to the random nature of the MDP, following the same policy may not result in the same trajectory.

Along the trajectory of an episode, an accumulation of discounted rewards from all states visited will be received. Our goal is to find the optimal policy as a sequence of actions a_{0}, a_{1}, a_{2}, ⋯ for a given MDP model of the environment in terms of its state transition dynamics and rewards, so that the value v(s_{0}) at the start state s_{0}, the expectation of the sum of all discounted future rewards, is maximized. This optimization problem can be solved by the method of dynamic programming, and such a process is called planning.

In an MDP, both the next state s′ and reward r are assumed to be random variables with joint probability p(s′, r|s, a) = P (s_{t+1} = s′, r_{t+1} = r|s_{t} = s, a_{t} = a) conditioned on the action a and the previous state s. The state transition probability becomes

p(s′|s, a) = P (s_{t+1} = s′|s_{t} = s, a_{t} = a) = ∑_{r∈R} p(s′, r|s, a)  (18.17)

and the expected reward r received after arriving at the current is

r(s, a) = E[r_{t+1}|s_{t} = s, a_{t} = a] = ∑_{r} r p(r|s, a) = ∑_{r} r ∑_{s′} p(s′, r|s, a) (18.18)

Following a specific random policy π(a|s), the agent takes an action a ∈ A(s) to transit from the current state s_{t} = s to the next state s_{t+1} = s′ with probability

p_{π}(s′|s) = P (s_{t+1} = s′|s_{t} = s) = ∑_{a∈A(s)} π(a|s) p(s′|s, a)  (18.19)

satisfying ∑_{s′} P_{π}(s′|s) = 1 and the reward in state s:

r_{π}(s) = E_{π}[r_{t+1}|s_{t} = s] = ∑_{a∈A(s)} π(a|s) r(s, a)  (18.20)

where E_{π} denotes the expectation with respect to a certain policy π(a|s). The summation over all available actions a ∈ A(s) in state s will be abbreviated to a in the following.

We further define two important functions with respect to a given policy π of an MDP:

• The state value function v_{π}(s) is the expected return at each state s of the MDP while following policy π, similar to the value function of an MRP as v(s) in Eq. (18.8), but now treated as a function of action a as well as state s:

v_{π}(s) = E_{π}[G_{t}|s_{t} = s]
= ∑_{a} π(a|s) (r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′))
= ∑_{a} π(a|s) q_{π}(s, a)  (18.21)

where q_{π}(s, a) is defined below.

• The state-action value function q_{π}(s, a) is the expected return of taking a specific action a (irrelevant to policy π) in state s, and then following π in all subsequent states:

q_{π}(s, a) = E_{π} [G_{t}|s_{t} = s, a_{t} = a]
= r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′)
= r(s, a) + γ ∑_{s′} P (s′|s, a) ∑_{a′} π(a′|s′) q_{π}(s′, a′)) (18.22)

where v_{π}(s′) is recursively represented as a weighted sum of q_{π}(s′, a′) based on Eq. (18.21).

The state-action value q_{π}(s, a) plays a more important role than v_{π}(s) as it allows the freedom of taking any action independent of a given policy π, thereby allowing the opportunity to improve an existing policy, e.g., by taking a greedy action to maximize the value q_{π}(s, a). The state-action value function q_{π}(s, a) is often abbreviated as the action value ore Q-value for convenience in future discussions.

As a function of the state-action pair (s, a), the state action value q_{π}(s, a) can be represented as a table of |S| rows each for one of the states s, and |A| columns each for one of the actions a. The Q-value for each state-action pair stored in the table can be updated iteratively by various algorithms for learning the Q-values.

In particular, if the policy is deterministic with π(a|s) = 1 and a = π(s), then we have

P_{π}(s′|s) = P (s′|s, a),  r_{π}(s) = r(s, a)  (18.23)

and the Bellman equations in Eqs. (18.21) and (18.22) become the same:

v_{π}(s) = r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′) = q_{π}(s, π(s))  (18.24)

same as the Bellman equation of an MRP in Eq. (18.8), and can be solved iteratively if γ < 1, same as in Eq. (18.12).

The figure above illustrates the Bellman equations in Eqs. (18.21) and (18.22), showing the bootstrapping of the state value function in the dashed box on the left, and that of the action value function in dashed box on the right. Specifically here the word bootstrapping means the iterative method that updates the estimated value at a state s based on the estimated value of the next state s′. After taking oen of the actions a ∈ A(s) available in state s based on policy π(a|s), an immediate reward r(s, a) is received. However, which state the MDP will transit into as the next state s′ is random depending on P (s′|s, a) of the environment but independent of the policy. This uncertanty is represented graphically by a dashed circle called a chance node associated with the action value q_{π}(s, a) as the sum of the immediate reward r(s, a) and the expected value of the next state, the weighted average of values v(s′) of all possible state s′ ∈ S.

18.3 Model-Based Planning

Given a complete MDP model in terms of p(s′, r|s, a) of the environment, the goal of model-based planning is to find the optimal policy π^{∗} that achieves the maximum value:

v^{∗}(s) = max_{π} v_{π}(s),  π^{∗}(s) = arg max_{π} v_{π}(s) ≥ π ∀π  (18.25)

based on the partial order that compares different policies according to the corresponding values:

If v_{π_{1}}(s) ≥ v_{π_{2}}(s) ≥ ⋯ v_{π_{k}}(s), ∀s ∈ S
Then π_{1} ≥ π_{2} ≥ ⋯ ≥ π_{k}  (18.26)

The problem of policy optimization can be addressed based on two subproblems: policy evaluation to measure how good a policy is, and policy improvement to get a better policy for higher values.

One way to solve this optimization problem is to use brute-force search to enumerate all |A(s)| available policies at each of the N = |S| states s ∈ S. If we assume the numbers of avilable actions in all states are the same for simplicity, i.e., A(s_{1}) = ⋯ = A(s_{N}) = A, then the computational complexity of this method is O(|A|^{|S|}), likely to be too high for such a brute-force search to be practical if N is large.

A more efficient and therefore more practical way to optimize the policy is to do it iteratively. Consider a simpler task of improving upon an existing deterministic policy π at a single transition step from the current state s to a next state s′. Instead of taking the action a = π(s) dictated by the policy, we can improve it by taking the greedy action that maximizes the action value function in Eq. (18.22), while subsequent steps still follow the old policy π. Now we get a new policy:

π′(s) = arg max_{a} q_{π}(s, a) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)] (18.27)

so that the action value is higher than that of the old one:

q_{π}(s, π′(s)) = max_{a} q_{π}(s, a) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)] ≥ q_{π}(s, π(s)) = v_{π}(s)

(18.28)

where the last equality is due to Eq. (18.24) for a deterministic policy.

While it is obvious that this greedy method will result in a higher return for the single state transition from s to s′, we can further prove the following policy improvement theorem stating that following this greedy method in all subsequent states, we will get a new policy π′ that achieves higher values than the those of the old one π at all states:

v_{π′}(s) ≥ v_{π}(s),  ∀s ∈ S  (18.29)

The proof is by recursively applying the greedy method to replace the old policy π(s) by the new one π′(s) for a higher value for each of the subsequent steps one at a time:

v_{π}(s) ≤ q_{π}(s, π′(s)) = max_{a} q_{π}(s, a) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)]
= r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s))v_{π}(s′)
≤ r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) max_{a′} q_{π}(s′, a′)

= r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) [r(s′, π′(s)) + γ ∑_{s′′} P (s′′|s′, π′(s′))v_{π}(s′′)]

= r(s, π′(s)) + γr(s′, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) [γ ∑_{s′′} P (s′′|s′, π′(s′))v_{π}(s′′)]
≤ ⋯
= r(s, π′(s)) + γr(s′, π′(s′)) + γ^{2}r(s′′, π′(s′′) + ⋯
= v_{π′}(s)  (18.30)

i.e., π′(s) ≥ π(s).

The optimal state value v^{∗}(s) and action value q^{∗}(s, a) can therefore be achieved by replacing the weighted average over all possible actions at each state (s and s′) in the Bellman equations in Eqs. (18.21) and (18.22) by their maximum:

v^{∗}(s) = max_{a} q^{∗}(s, a) = max_{a} (r(s, a) + γ ∑_{s′} P (s′|s, a)v^{∗}(s′))  (18.31)

and

q^{∗}(s, a) = r(s, a) + γ ∑_{s′} P (s′|s, a)v^{∗}(s′) = r(s, a) + γ ∑_{s′} P (s′|s, a) max_{a′} q^{∗}(s′, a′)

(18.32)

Eqs. (18.31) and (18.32) are the Bellman optimality equations, which should hold for all N states in S = {s_{1}, ⋯, s_{N}} and can be written in vector form:

v^{∗} = max_{a} (r_{a} + γP_{a}v^{∗})  (18.33)

where

v^{∗} = [ v^{∗}(s_{1}) ; ⋮ ; v^{∗}(s_{N}) ] , r_{a} = [ r(s_{1}, a) ; ⋮ ; r(s_{N}, a) ] , P_{a} = [ \begin{matrix}P (s_{1}|s_{1}, a) & ⋯ & P (s_{N}|s_{1}, a) \\ ⋮ & ... & ⋮ \\ P (s_{1}|s_{N}, a) & ⋯ & P (s_{N}|s_{N}, a) \end{matrix} ]

(18.34)

This nonlinear equation system can be solved iteratively to for v^{∗}:

v_{n+1}^{∗} = max_{a} (r_{a} + γP_{a}v_{n}^{∗}) = B(v_{n})  (18.35)

This iteration is to be used for policy evaluation in the algorithm value iteration below for finding the optimal policy. This iteration will converge as function B(v) is a contraction mapping with γ < 1:

||B(v_{i}) − B(v_{j})|| = || max_{a_{i}} (r_{a_{i}} + γP_{a_{i}}v_{i}) − max_{a_{j}} (r_{a_{j}} + γP_{a_{j}}v_{j}) ||
≤ || max_{a_{i}} (r_{a_{i}} + γP_{a_{i}}v_{i} − r_{a_{i}} − γP_{a_{i}}v_{j}) ||
= max_{a_{i}} γ||P_{a_{i}}(v_{i} − v_{j})|| ≤ max_{a_{i}} γ||P_{a_{i}}|| ||v_{i} − v_{j}||
= γ||v_{i} − v_{j}|| < ||v_{i} − v_{j}||  (18.36)

where ||P|| = ||P||_{∞} = 1 is the p-norm (p = ∞) of a stochastic matrix, which is known to be 1.

Given the complete information of the MDP model in terms of p(s′, r|s, a), we can find the optimal policy π^{∗} using either policy iteration (PI) or value iteration (VI) based on the DP method.

• Policy Iteration

Carry out the following two processes iteratively from some arbibrary initial policy π until convergence when π no longer changes:

– Policy evaluation:

Given a deterministic policy π, Eq. (18.21) becomes the Bellman equation in Eq. (18.24), a linear equation system of N = |S| equations of N unknowns v_{π}(s_{1}), ⋯, v_{π}(s_{N}), which can be solved iteratively as in Eq. (18.12) to find values at all states: v_{n+1}^{π}(s) = r(s, π(s)) + γ ∑_{s′} P (s′|s, π(s))v_{n}^{π}(s′) ∀s ∈ S

– Policy improvement:

Given v_{π}(s) based on policy π, get a better policy π′ by the greedy method as in Eq. (18.27): π′(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, π(s))v_{π}(s′)] ∀s ∈ S We see that in each round of the policy iteration, two intertwined and interacting processes, the policy evaluation and policy improvement, are carried out alternatively, one depending on the other, and one completing before the other starts. The value functions v_{π}(s) at all states are sysmeticaly evaluated before they are updated, and the policy π(a|s) at all states is updated before it is evaluated. While these two processes chase each other as moving target iteratively and eventually converge to the optimal policy π^{∗} achieving the optimal value v^{∗}. This approach is said to be synchronous, as shown here:

π_{o} ⟶^{E} v_{π_{0}} ⟶^{I} π_{1} ⟶^{E} v_{π_{1}} ⟶^{I} ⋯ ⟶^{I} π^{∗} ⟶^{E} v_{π^{∗}} = v^{∗}  (18.37)

The pseudo code for the algorithm is listed below: Initialize:

v(s) ∈ ℝ, v(s) = 0 if s is terminal node, π(s) ∈ A(s), ∀s ∈ S; Set tolerence tol (small positive value);

repeat

Policy Evaluation (find v(s) ∀s ∈ S):

repeat

δ = 0
for ∀s ∈ S:
v′(s) = r(s, π(s))+γ ∑_{s′} P (s′|s, π(s))v(s′)
δ = max(δ, |v(s) − v′(s)|)

until δ < tol Ploicy Improvement (by taking greedy action):

Done = T for each s ∈ S

π′(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, π(s))v(s′)]

if π′(s) ≠ π(s), Done = F

end for

until done

• Value Iteration:

One drawback of the policy iteration method is the high computational complexity of the inner iteration for policy evaluation inside the outer iteration for policy evaluation. To speed up this process, we could terminate the inner iteration early before convergence. In the extreme case, we simply combine policy evaluation and improvement into a single step, resulting in the following value iteration contains two steps:

– Find the optimal values:

Based on some initial value v_{0}(s), solve the Bellman optimality equation in Eq. (18.31) by iteration in Eq. (18.35):

v_{n+1}(s) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{n}(s′)] ∀s ∈ S (18.38)

– Find the optimal policy:

Find π^{∗}(s) that maximizes v^{∗}(s):

π^{∗}(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{n}(s′)] ∀s ∈ S

(18.39)

Here is the pseudo code for the algorithm: Initialize: v(s) = 0 ∀s ∈ S repeat

δ = 0;

for each s ∈ S

v′(s) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v(s′)]
δ = max(δ, |v′(s) − v(s)|)
v(s) = v′(s)

end for until δ < tol π^{∗}(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v(s′)]

In the PI method considered above, the value functions at all states are evaluated before the policy is updated for all states, as illustrated below:

This synchronous policy iteration can be modified so that the values at some or even one of the states are evaluated in-place before other state values are updated, and the policy at some or even one of the states are updated in-place before that at other states, resulting in an asynchronous method called general policy iteration (GPI), to be discussed in detail in the next sections on model-free control.

18.4 Model-Free Evaluation and Control

The previously discussed dynamic programming methods find the optimal policy based on the assumption that the MDP model of the environment is completely available, i.e., the dynamics of MDP in terms of its state transition and reward mechanism are known. A given policy π can then be evaluated based the transition probabilities p(s′|s, a), and improved based on greedy action selection at each state. Such a model-based optimization problem, called planning, involves no learning of the environment.

Now we consider the optimization problem with an unknown MDP model of the environment. In this model-free problem, called control, the state transition and reward probabilities of the MDP model are unknown, the value functions can no longer be calculated directly based on Eqs. (18.21) and (18.22) as in modelbased planning. Instead, now we need to learn the MDP model by sampling the environment, i.e., by repeatedly running a large number of episodes of the stochastic process of the environment while following some given policy, and estimate the value functions as the average of the actual returns received during the sampling process. This method is generally referred to as Monte Carlo (MC) method. At the same time, the given policy can also be improved to gradually approach optimality.

This approach is called general policy iteration (GPI), as illustrated in Eq. (18.40) below, similar to the policy iteration (PI) algorithm for model-based planning illustrated in Eq. (18.37). In general, all model-free control algorithms are based on GPI by which the two alternating processes of policy evaluation and policy imprvement are carried out iteratively. This GPI can also be similarly illustrated in Fig. ?? for the GI.

π_{o} ⟶^{E} q_{π_{0}} ⟶^{I} π_{1} ⟶^{E} q_{π_{1}} ⟶^{I} ⋯ ⟶^{I} π^{∗} ⟶^{E} q_{π^{∗}} = v^{∗}  (18.40)

While GPI and GI illustrated in respectively in Eqs. (18.40) and (18.37) may look similar to each other, there are some essential differences between the two, as listed below.

• First, in model-free control the policy improvement is based on the actionvalue function q_{π}(s, a), instead of the state value function v_{π}(s) in modelbased planning. This is because without the a specific MDP model the state value function can no longer be calculated, while the action value function can still be estimated based on the actual rewards received while sampling the environment. Similar to how we improve the policy by taking a greedy action to achive a higher state value in model-based planning, here we improve the policy by taking an action different from that dictated by the given policy π to achieve a higher action value based on some ϵ-greedy method, as discussed below.

• Second, as estimating the action value by the MC method requires running a large number of episodes while the sampling of the environment, the computational complexity for policy evaluation in the model-free case is much higher than that in model-based planning. To speed up this process in model-free control, the iterative policy evaluation is no longer fully carried for it to converge to the true action value of the policy. Instead, in modelfree control, the iterative update of the action value is carried out only once, followed immediately by the next phase of policy improvement. Such an estimated action value is innevitably very noisy, but due to the much reduced complexity, we can afford to run a large number of sample episodes while sampling the envirionment.

• Third, in the PI process of model-based planning, the action value is found based on all available actions at each state (Eq. (18.22)) during policy evaluation, and we only need to exploit the greedy action at each state to find the policy that achieves the maximum action value q_{π}(s) (Eq. (18.27)) during policy improvement. However, here in model-free control, we have to learn the action value q_{π}(s, a), as a function of action a ∈ A(s) as well as state s by sampling the environment. Such an estimated action value based only on some partial sample data is inevitably noisy, especially in the early stage of learning when many of the states have not yet been visited yet. We therefore need to explore all actions at each state to better learn the value function, as well as to exploit the greedy action to improve the policy.

This issue of exploration versus exploitation can be addressed by the ϵ-greedy method to make a proper trade-off between the exploitation of the greedy action and the exploration of other non-greedy actions, so that the agent can be exposed to all possible state-action pairs and gradually learn the action value function while at the same time also improve the policy being followed. In this method, we define ϵ ∈ [0, 1] as the probability to explore any randomly chosen action a ∈ |A(S)| out of all actions available in state s, each with an equal probability ϵ/|A(s)|, and 1 − ϵ is the probability of taking the greedy action as one of the |A(s)| actions, with probability 1 − ϵ + ϵ/|A(s)|.

π′(a|s) = \{ \begin{matrix}arg max_{a} q(s, a) & P r = 1 − ϵ + ϵ/|A(s)| \\ any a ∈ A(s) & P r = ϵ/|A(s)| \end{matrix}  (18.41)

For example, if |A(s)| = 4 and ϵ = 2/3, the greedy action may be chosen with probability 1 − ϵ + ϵ/|A(s)| = 3/6, while each of the remaining three non-greedy actions may be chosen with probability ϵ/|A(s)| = 1/6. Such a policy describes the behavior of the agent, and is therefore called behavior policy, different from the policy being followed.

We further note that a larger value of ϵ (close to 1) can be used to emphasize exploration at the early stage of the iteration when many state-action pairs have not been visited yet, while progressively smaller ϵ values (approaching to 0) can be used to amphasize exploitation for higher values in the later stage of the iteration when the action value function has been more reliably learned. For example, we can let ϵ_{k} = 1/k be inversely proportional to the current number of iterations k, so that the larger ϵ values at the early stage of the iteration allow more sate-action pairs to be explored, while smaller ϵ values in the later stage of the iteration encourage exploitation of the greedy actions. To the limit k → ∞ and ϵ ⟶ 0, the ϵ-greedy method approaches absolute greedy method.

Similar to how we proved the policy improvement theorem in Eq. (18.30) stating that the greedy policy π′(s) achieves state value v_{π′}(s) no lower than v_{π}(s) achieved by the original policy π, i.e., π′(s) ≥ π(s), here we can also prove the same policy improvement theorem stating that the ϵ-greedy policy π′ achieves action value q_{π′}(s, a) no lower than q_{π}(s, a) by the original policy π:

q_{π}(s, π′(s)) = ∑_{a} π′(a|s)q_{π}(s, a)
= \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + (1 − ϵ) max_{a} q_{π}(s, a)
≥ \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + (1 − ϵ) ∑_{a} \frac{π(a|s) − ϵ/|A(s)|}{1 − ϵ}q_{π}(s, a) (18.42)

where the inequality is due to the fact that the maximum value of q_{π}(s, a) is no less than the average of action values q_{π}(s, a) over all a ∈ A(s) weighted by some normalized coefficients adding up to 1:

∑_{a} \frac{π(a|s) − ϵ/|A(s)|}{1 − ϵ} = \frac{1}{1 − ϵ} ∑_{a} (π(a|s) − \frac{ϵ}{|A(s)|}) = \frac{1}{1 − ϵ}(1 − ϵ) = 1

(18.43)

Continuing the equation above we further get

q_{π}(s, π′(s)) ≥ \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) − \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + ∑_{a} π(a|s)q_{π}(s, a)
= ∑_{a} π(a|s)q_{π}(s, a) = v_{π}(s)  (18.44)

The iterative process of the GPI method for model-free control is illustrated in the figure below, similar to the PI method for model-based planning based planning illustrated in Fig. ??, but with the state value v_{π}(s) replaced by the action value q_{π}(s, a), and the greedy method replaced by ϵ-greedy method.

Also, here the estimated action value q̃ is obtained with only one iteration for policy evaluation, instead of a much more accurate estimate that could otherwise be obtained if the iterative evaluation was fully carried out to its convergence (the top line). At the same the time, based on the estimated q̃ policy improvement is carried out by the ϵ-greedy method (the bottom line).

The main task in model-free control is to evaluate the state and action-value functions given in Eqs. (18.21) and (18.22) while following a given policy π without a specific MDP model. This is done by sampling of the dynamic process of the environment, and then estimating the value functions as the average of the actual returns received by the agent. Specifically, we run a large number of episodes (all assumed to terminate) of the MDP model of the environment by taking actions a_{t} = π(s_{t}) in each state s_{t}, (t = 1, ⋯, T):

s_{0} → (a_{1}, r_{1}, s_{1}) → (a_{2}, r_{2}, s_{2}) → ⋯ → (a_{T} , r_{T} , s_{T})  (18.45)

and estimate the value functions based on the averaged returns G_{t} actually received from these episodes.

Such a sequence of states visited is called a trajectory, and the trajectories of different episodes are in general different from each another due to the random nature of the environment. In the following sections we will consider specific algoithms for the implementation of the general model-free control discussed above.

To prepare for specific discussion of the GPI methods, we first consider a general problem of the estimation of the value of a random variable x as the running average of its samples beging collected in real time. Speicically the average x̂_{n} based on previous n samples x_{1}, ⋯, x_{n} is updated incrementally upon receiving a new sample x_{n+1}:

x̂_{n+1} = \frac{1}{n + 1} ∑_{i=1}^{n+1} x_{i} = \frac{1}{n + 1} (x_{n+1} + ∑_{i=1}^{n} x_{i})
= \frac{1}{n + 1}(x_{n+1} + nx̂_{n}) = x̂_{n} + \frac{1}{n + 1}(x_{n+1} − x̂_{n})  (18.46)

This running average can be generalized to

x̂_{n+1} = x̂_{n} + α(x_{n+1} − x̂_{n})  (18.47)

This equation can be interpreted as

newEstimate = oldEstimate + stepSize × (target − oldEstimate)
= oldEstimate + stepSize × error
= oldEstimate + increment  (18.48)

and considered as one iterative step in the method of stochastic gradient descent (SGD), for minimizing the squared error ε = (x_{n+1} −x̂_{n})^{2}/2 between the old average x̂_{n} and the latest sample x_{n+1}, and the second term is the gradient dε/dx̂_{n} weighted by the step size α ∈ [0, 1], which controls how samples are weighted differently. In the extreme case when α = 1, x̂_{n} = x_{n} with no contribution from any previous samples; on the other hand when α is close to 0, x̂_{n}, the most recent samples has little contribution. We can therefore properly adjust α to fit our specific need. For example, if we gradually reduce α, then the estimated average will become stabilized while enough samples have been collected. On the other hand, if we let 0 < α < 1/(n + 1), the more recent samples are weighted more heavily than the earlier ones. Such a stratigy is suitable when estimating parameters of a nonstationary system, such as an MDP with varying behaviors when the policy is being modified continuously.

Specifically in model-free control Eq. (18.47) can be used to iteratively update the estimated value functions while sampling the environment. Both the state and action values as the expected return are estimated as the average of all previous sample returns, and they are updated incrementally when a new sample return G_{n+1}, the target, becomes available:

v_{n+1}(s) ⇐ v_{n}(s) + α(G_{n} − v_{n}(s))
q_{n+1}(s, a) ⇐ q_{n}(s, q) + α(G_{n} − q_{n}(s, a))  (18.49)

as we will see in the following sections.

18.4.1 Monte Carlo (MC) Algorithms

We first consider a simple problem of evaluating an existing policy π in terms of its value function v_{π}(s) = E[ G_{t} ] at state s = s_{t}, the expectation of return G_{t}, which can be obtained in the model-based case by the Bellman equation in Eq. (18.24):

v_{π}(s) = r(s, π(s)) + γ ∑_{s′} p(s′|s, π(s))v_{π}(s′)  (18.50)

based on v_{π}(s′) of all possible next states each weighted by the corresponding transition probability p(s′|s, a), which is no longer available now in the model-free case. We can still find the value function based on the MC method by running multiple sample episodes in the environment, and estimate v_{π}(s) as the average of the actual returns G_{t}, which in turn can be calculated as the sum of discounted rewards r_{t+1}, ⋯, r_{T} from the current state s = s_{t} onward to the terminal state s_{T}, found at the end of the episode (assumed to have finite horizon).

This method has two versions, first-visit and every-visit, depending on whether only the first or every visit to a state in the trajectory of an episode is counted in calculation of the return. The pseudo code below is for first-visit version due to the if statement, which can be removed for the every-visit version.

Input: a given policy π to be evaluated Initialize: G_{t} = [ ] (empty lists) for all t = 0, ⋯, T − 1 loop (for each episode)

run episode to the end while following π to get sample data: (s_{0}, a_{0}, r_{1}), (s_{1}, a_{1}, r_{2}), ⋯, (s_{T −1}, a_{T −1}, r_{T})

G = 0
for t = T − 1, T − 2, ⋯, 0 (for each step)
G = γG + r_{t+1}
if S_{t} ∉ {S_{0}, ⋯, S_{t−1}} (first visit)
G_{t} = [G_{t}, G] (append G to list G_{t})
V (S_{t}) = average(G_{t}), ∀t = 0, , ⋯, T − 1

In the every-visit case, all sample returns G_{t} from multiple visits to state s_{t} are used, although they may be correlated and not independent of each other, i.e., they are not necessarily i.i.d. samples and their average may be biased. This problem can be avoided in the first-visit case, where only the returns of the first visit to each state are counted as independent samples drawn from the same distribution, and their average is not biased. However, as only a fraction of the sample points collected is used in the calculation of the returns, the cost of this unbiased estimation is its high variance, which can be reduced only if a large number of episodes are used to get enough samples for a more statistically reliable average. This is the typical trade-off between variance and bias errors.

In the code above, the function average finds the average of all elements of a list, only when all sample returns are available at the end of each episode. A more computationally efficient method (in terms of both space and complexity) is to find the average incrementally as in Eq. (18.49). Based on this incremental average, policy evaluation in terms of both the state and action values can also be carried out by the code below:

Initialize v(s) = 0, ∀s loop (for each episode)

run episode following π to get G_{t}, ∀t for t = 0, ⋯, T − 1 (for each step)

if s_{t} is visited the first time v(s_{t}) = v(s_{t}) + α(G_{t} − v(s_{t}))

and

Initialize: q(s, a) = 0, ∀(s, a) loop (for each episode)

run episode following π to get G_{t}, ∀t

for all (s, a) pairs visited

if (s, a) is visited the first time q(s, a) = q(s, a) + α(G_{t} − q(s, a))

The if condition can be removed for every-visit version of the algorithm.

We next consider the MC algorithm for model-free control, based on the generalized policy iteration of alternating and interacting policy evaluation and policy improvement. Specifically, while sampling the environment by following an existing policy π the action value q_{π}(s), instead of the state value v_{π}(s), is gradually learned and the policy is gradually improved at the same time. This iteration will converge to the optimal policy at the limit when the number of iterations goes to infinity.

The pseudo code for this lgorithm is listed below.

Input: ϵ > 0, α > 0, and a policy π Initialize: q(s, a) = 0, ∀(s, a), π = ϵ − soft (arbitrarily) loop (for each episode)

run episode following π to get G_{0}, ⋯, G_{T −1}

for t = 0, ⋯, T − 1

if (s_{t}, a_{t}) is visited the first time q(s_{t}, a_{t}) = q(s_{t}, a_{t}) + α(G_{t} − q(s_{t}, a_{t}))

a^{∗} = arg max_{a} q(s_{t}, a_{t})
for ∀a ∈ A(s_{t})

π(a|s_{t}) = \{ \begin{matrix}1 − ϵ + ϵ/|A(s_{t})| & if a = a^{∗} \\ ϵ/|A(s_{t})| & else \end{matrix}

Here the ϵ-greedy policy π(a|s) is based on action value function q(s, a) (maximized by the greedy action). Through out the iteration, the policy is modified following the action value function, while at the same time the action value function is modified based on the policy, until the process converges to the desired optimality.

We note that the optimal policy π(s) obtained by the algorithm above is not completely deterministic due to the ϵ-greedy approach, but it is said to be neardeterministic as the greedy action is favored over other non-greedy actions.

18.4.2 Temporal Difference (TD) Algorithms

The temporal difference (TD) method is a combinaion of the MC method considered above and the bootstrapping DP method based on the Bellman equation. The main difference between the TD and MC methods is the target, the return G, in the incremental average in Eq. (18.49) for estimating either the state or action value functions. While in the MC method the target is the actual return G_{t} calculated at the end of the episode when all subsequent rewards r_{t+1}, ⋯, r_{T} are available, here in the TD method the target, called TD target, is the sum of the immediate reward r_{t+1} and the previouosly estimated value v_{π}(s_{t+1}) at the next state:

G_{t} = r_{t+1} + γv_{π} (s_{t+1})  (18.51)

We see that the TD method is an in-place bootstrapping method, and it makes more efficient use of the sample data and updates more frequently the value functions being estimated and the policy being improved at every step of an episode, instead of at the end of the episode as in the MC method. Also, it can be used even if the dynamic process of the environment is non-episodic with infinite horizon.

Again, we first consider the simpler problem of policy evaluation. As in the MC method, we estimate the value function v_{π}(s) as the average of the actual returns G found by running multiple episodes while sampling the environment. Substituting this TD target into the running average in Eq. (18.49) for the value function above, we get

v_{π}(s) ⇐ v_{π}(s) + α(G − v_{π}(s))
= v_{π}(s) + α(r + γv_{π}(s′) − v_{π}(s)) = v_{π}(s) + αδ  (18.52)

where δ, called the TD error, is the difference between the TD target and the previous estimate:

δ = G − v_{π}(s) = r + γv_{π}(s′) − v_{π}(s)  (18.53)

and v_{π}(s′) is the current estimatee of the value of the next state (bootstrapping). It can be shown that the iteration of this TD method converges to the true value v_{π}(s), if the step size is small enough.

Here is the pseudo code for policy evaluation using the TD method, based on parameters γ and α:

Input: a given policy π to be evaluated Initialize: v(s) for all s ∈ S arbitrarily, v(s) = 0 for terminal state s, α ∈ (0, 1] loop (for each episode)

s = s_{0}

while s is not terminal (for each step) take action a = π(s), get reward r and next state s′ v(s) = v(s) + α[r + γv(s′) − v(s)] s = s′

As the TD algorithm updates the value function at every step of the episode, it uses the sample data more frequently and efficiently, and therefore has lower variance error than the CM method that updates the estimated value function at the end of each episode. On the other hand, the TD method may be biased when compared to the first-visit MC method, due to the arbitrary initialization of the value functions.

Based on the TD method for model-free policy evaluation, we now further consider the TD method for model-free control to gradually learn the optimal policy by updading the Q-values of all state-action pairs (s, a) estimated iteratively as the running average of the sample Q-values at each time step of an episode while sampling the environment.

The Q-values for all state-action pairs can be stored in a state-action table which is iteratively updated by running many episodes of the unknown MDP by the TD method to gradually approach the maximum Q-values achievable by taking each action at each state, i.e., the optimal action is the one with the highest Q-value.

This method has two different flavors, the on-policy algorithm, which updates the Q-value by following the current policy such as an ϵ-greedy policy, and the off-policy algorithm, which updates the Q-value by taking actions different from the current policy such as the greedy action. In particular, if the policy currently being followed is greedy (instead of ϵ-greedy), the two algorithms are the same.

• State-Action-Reward-State-Action (SARSA)

The Q-value q(s, a) of each state-action pair (s, a) is estimated as the running average of the return G, the sum of the immmediate reward r and the action value q(s′, a′) of the next state based on action a′ dictated by the current policy (e.g.,ϵ-greedy):

q(s, a) ⇐ q(s, a) + α(G − q(s, a))
= q(s, a) + α(r + γq(s′, a′) − q(s, a))  (18.54)

This algorithm is called SARSA, as it updates the Q-value based on the current state s and action a, the immediate reward r, and the next state s′ and action a′. The pseudo code of SARSA algorithm is listed below. Initialize: q(s, a) = 0 for all (s, a), α ∈ (0, 1], ϵ > 0, denote ϵ-greedy policy

pi(a|s) by π loop (for each episode) s = s_{0} get action a according to π based on q(s, a) while s is not terminal (for each step)

take action a, get reward r and next state s′ get action a′ according to π based on q(s′, a) q(s, a) = q(s, a) + α[r + γq(s′, a′) − q(s, a)] s = s′, a = a′

As a variation of SARSA, the expected SARSA updates the Q-value at state s based on the expected Q-value of the next state s′, the weighted average of the Q-values resulting from all possible actions, instead of only one action. Consequently, the estimated Q-values by the expected SARSA have lower variance than SARSA, and a higher learning rate α can be used to speed up the learning process.

q(s, a) = q(s, a) + α (r + γ ∑_{a} π(a|s′)q(s′, a) − q(s, a))  (18.55)

• Q-Learning

Same as SARSA, the Q-learning algorithm also estimates the Q-value q(s, a) of each state-action pair (s, a) as the running average of the return G, the sum of the immmediate reward r and the action value q(s′, a′) of the next state based on the greedy action a′ that maximizes the next state value q(s′, a′), different from that dictated by the current policy.

q(s, a) ⇐ q(s, a) + α(r + γ max_{a′} Q(s′, a′) − q(s, a))  (18.56)

The pseudo code of SARSA algorithm is listed below. Initialize: q(s, a) ∀s ∈ S, ∀a ∈ A(s) arbitrarily (q(s, a) = 0 for terminal s),

α ∈ (0, 1], ϵ > 0 loop (for each episode)

Initialize state s

while s is not terminal (for each step)

take action a according to π based on q(s, a), get reward

r and next state s′ q(s, a) = q(s, a) + α[r + γ max_{a′} q(s′, a)′ − Q(s, a)]

s = s′

Here is the comparison of the MC and TD methods in terms of their pros and cons:

• The MC method estimates the state value v_{π}(s_{t}), the expected return, by the true return G_{t} obtained at the end of the episode, i.e., the estimated value is updated once every episode;

The TD method estimates v_{π}(s_{t}) as the sum of the immediate reward r_{t+1} and the estimated state value at the next state v_{π}(s′), i.e., it is a bootstrap method, and the estimated value is updated at every step of every spisode.

• MC can only learn episodic (terminating) environments with complete episodes; while TD can learn continuing (non-terminating) environent of incomplete episodes.

• MC estimates v_{π}(s_{t}) is based on sample returns G_{t}, and it is unbiased, while TD uses the bootstrap approach to find the TD target based on sample data that are not necessarily i.i.d., and it is more sensitive to the initial guess of the value functions, it is more biased.

• MC is based on the sample returns G_{t} affected by many random events (state transitions, actions, and rewards), and in particular the first-visit version of the MC method only makes use of the sample data from the first visit of a state, it does not use the available data efficiently and it has high variance, while on the other hand the TD method is based on only one random variable, the estimated return, and it makes more frequent and efficient use of the sample data, it has lower variance.

• In the MC method, especially the first-visit version of it, the estimated value functions is unbiased; on the other hand, as the TD method is based on the bootstrap strategy and relies more strongly on the initial guess of the value functions being estimated, it tends to be more biased.

Here is a summary of the dynamic programming (DP) method for modelbased planning, and the Monte-Carlo (MC) and time difference (TD) methods for model-free control:

18.4.3 TD(λ) Algorithm

The MC and TD methods considered previously can be unified by the n-step TD(λ) method that spans a spectrum of which the MC and TD methods are two special cases at the opposite extremes.

Recall that both MC and TD algorithms updates iteratively the value functions being estimated and the policy being improved based on Eq. (18.49), but they estimate the target G_{t} in the equation differently. In an MC algorithm, the target is the actual return G_{t} = r_{t+1} + γr_{t+2} + ⋯ + γ^{T −t−1}r_{T}, the sum of discounted rewards of all future steps upto the terminal state s_{T}, available only at the end of each episode. On the other hand, in a TD algorithm, the target is G_{t} = r_{t+1} + γv_{π}(s)t + 1) the sum of the immeidate reward r_{t+1} available at each step of the episode, and the discounted value of the next state, based on bootstrapping. We therefore see that an MC algorithm updates the value functions and policy once every episode, while a TD algorithm updates once every step in the spisode.

As a trade-off between the MC and TD methods, the n-step TD algorithm can be considered a generalization of the TD method, where the target in Eq. (18.49) is an n-step return, the sum of the discounted rewards in the n subsequent states and the discounted value function at the following state s_{t+n} with 1 ≤ n < ∞:

G_{t:t+n} = r_{t+1} + γr_{t+2} + ⋯ + γ^{n−1}r_{t+n} + γ^{n}v_{π}(s_{t+n})  (18.57)

Specially, when • n = 1: the 1-step return is the sum of the immediate reward and the estimated

value of the next state, the same as the TD target in the TD method:

G_{t:t+1} = r_{t+1} + γv(s_{t+1})  (18.58)

• n = T − t: the n-step return is the sum of the discounted rewards from all future states upto the terminal state at the end of the episode, i.e., it is the return r_{t} defined in Eq. (18.7), the same as in the MC method:

G_{t:t+n} = r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ + γ^{T −t−1}r_{T} + v_{π}(s_{t+n}) = G_{t} (18.59)

where the value function of the terminal state v_{π}(s_{t+n}) = v_{π}(s_{T}) = 0 is zero.

• T − t < n < ∞: All states s_{n} beyond the terminal state s_{T} with n > T − t remain the same as the terminal state with value v(s_{T}) = 0 and return G_{t}, same as in the MC method.

We see that all these n-step returns form a spectrum with the MC and TD methods at the two ends.

Based on these n-step returns of different n values, the n-step TD algorithm can be further generalized to a more computationally advantageous and therefore more useful algorithm called TD(λ), by which the MC and TD algorithms are again unified as two special cases at the opposite ends of a spectrum.

We first define the λ-return G_{t}^{λ} as the weighted average of all n-step returns for n = 1, 2, ⋯, ∞:

G_{t}^{λ} = (1 − λ) ∑_{n=1}^{∞} λ^{n−1}G_{t:t+n}  (18.60)

where 0 ≤ λ < 1. Note that the weights decay exponentially, and they are normalized due to the coefficient 1 − λ:

(1 − λ) ∑_{n=1}^{∞} λ^{n−1} = (1 − λ) ∑_{n=0}^{∞} λ^{n} = \frac{1 − λ}{1 − λ} = 1  (18.61)

The λ-return G_{t}^{λ} defined above can be expressed in two summations, the sum of the first T − t − 1 n-step returns G_{t:t+1}, ⋯, G_{t:t+T −1}, and the sum of all subsequent n-step returns G_{t:T} = ⋯ = G_{t:∞} = G_{t}:

G_{t}^{λ} = (1 − λ) ∑_{n=1}^{T −t−1} λ^{n−1}G_{t:t+n} + (1 − λ) ∑_{n=T −t}^{∞} λ^{n−1}G_{t}

= (1 − λ) ∑_{n=1}^{T −t−1} λ^{n−1}G_{t:t+n} + λ^{T −t−1}G_{t},  (0 ≤ t ≤ T) (18.62)

where the coefficient of G_{t} in the second term is the sum of coefficients in the second summation:

(1 − λ) ∑_{n=T −t}^{∞} λ^{n−1} = (1 − λ) ∑_{m=0}^{∞} λ^{m}λ^{T −t−1} = λ^{T −t−1}  (18.63)

where m = n − T + t. We see that in Eq. (18.62) all n-step returns G_{t:t+n} are weighted by exponentially decaying coefficient λ^{n−1}, except the true return G_{t} which is weighted by λ^{T −t−1}.

Again consider two special cases:

• λ = 0: all terms in Eq. (18.62) are zero, except the first one with n = 1:

G_{t}^{λ=0} = λ^{n−1}G_{t:t+n}|_{n=1} = 0^{0} G_{t:t+1} = r_{t+1} + γv(s_{t+1})  (18.64)

This is simply the TD target, i.e., TD(0) is the TD method.

• λ = 1: the first summation is zero and

G_{t}^{λ=1} = G_{t}  (18.65)

i.e., TD(1) is the MC method.

We therefore see that TD(λ) is a general algorithm of which the two special cases TD(0) and TD(1) are respectively the TD and MC methods.

In Eq. (18.62), G_{t}^{λ} is calulated as the weighted sum of all n-step returns G_{t:t+n} upto the last one G_{t} available at the end of the episode. This summation can be truncated to include fewer terms before reaching the terminal state at the end of the episode:

G_{t:h}^{λ} = (1 − λ) ∑_{n=1}^{h−t−1} λ^{n−1}G_{t:t+n} + λ^{h−t−1}G_{t:h},  (0 ≤ t < h ≤ T) (18.66)

Specially when h = T, G_{t:h}^{λ} = G_{t}^{λ} same as before, and when h = t + 1, G_{t:h}^{λ} = G_{t:t+1} = r_{t+1} + γv(s_{t+1}), same as the TD target.

Based on λ-return, the iterative update of the value function v_{π}(s_{t}) in Eq. (18.52) can be modified to:

v_{π}(s_{t}) = v_{π}(s_{t}) + α(G_{t}^{λ} − v_{π}(s_{t})) = v_{π}(s_{t}) + αδ_{t}  (18.67)

where

δ_{t} = G_{t}^{λ} − v_{π}(s_{t})  (18.68)

This iteration is called the forward view of TD(λ), which is similar to the MC method, as they are based on either G_{t} or G_{t}^{λ}, available only at the end of each episode.

An alternative method is the backward view of TD(λ), which can be shown to be equivalent to the forward-view of TD(λ), but it is similar to the TD(0) method, as they are both based on the immediate reward r_{t+1} available at each time step of the episode, and therefore more computationally convenient.

Specifically, the backward view of TD(λ) is based the eligibility trace e_{t}(s) for each of the states, which decays exponentially upon each state transition

e_{t}(s) = γλe_{t−1}(s) (∀s ∈ S)  (18.69)

but is boosted by an increment 1 to become e_{t}(s) = e_{t}(s)+1 if s = s_{t} is currently visited.

Now the iterative update of the estimated value function in Eq. (18.52) is modified so that all states, instead of only the one currently visited, are updated, but to different extents based on e_{t}(s):

v_{π}(s) = v_{π}(s) + αδ_{t}e_{t}(s),  ∀s ∈ S  (18.70)

where the TD error δ_{t} is the same as in TD(0), given in Eq. (18.53):

δ_{t} = r_{t+1} + γv_{π}(s_{t+1}) − v_{π}(s)  (18.71)

The eligibility trace e_{t}(s) as defined above is motivated by the frequency and recency of the visites to each state. If a state s has been more frequently and recently visited compared to others, its e_{t}(s) is greater than others and its value function v_{π}(s) will be updated by a greater increment than others.

Here is the pseudo code for the backward view of the TD(λ) methd for policy evaluation:

Input: policy π to be evaluated Initialize: v(s) = 0 ∀s ∈ S loop (for each episode)

s = s_{0}

e(s) = 0, ∀s ∈ S

while s is not terminal (for each step) take action a based on π(a|s), get reward r and next state s′ δ = r + γv(s′) − v(s) e(s) = e(s) + 1 for all s

v(s) = v(s) + αδe(s)
e(s) = γλe(s)

s = s′

Note that values at all states are updated, but to different extents depending on e(s), different from the TD algorithm where only the value at the state currently visited is updated.

Again consider two special cases:

• λ = 0, e(s) = 0 for all states except the current s being visited, i.e., TD(λ) becomes TD(0), the same as the TD method.

• λ = 1, then e(s) is scaled down by a factor γ < 1 for all states except the current s being visited, i.e., TD(λ) becomes TD(1), the same as the MC method. However, different from the MC method that updates the value function v(s) at the end of each spisode, here v(s) is still updated at every step of the episode due to the backward view of the method.

This backward view of the TD(λ) method can be applied to model-free control when the state value function is replaced by the action value function. Corresponding to the SARSA and Q-learning algorithms based on TD(0) in the previous section, here are the two algorithms based on eligibility traces:

• SARSA(λ) algorithm:

Initialize: q(s, a) = 0, ∀(s, a), α ∈ (0, 1], ϵ > 0, denote ϵ-greedy policy

pi(a|s) by π loop (for each episode)

s = s_{0} e(s, a) = 0, ∀(s, a) get action a according to π based on q(s, a) while s is not terminal (for each step)

e(s, a) = e(s, a) + 1 take action a, get reward r and next state s′ get action a′ according to π based on q(s′, a) δ = r + γq(s′, a′) − q(s, a) for all (s, a)

q(s, a) = q(s, a) + αδ e(s, a)

e(s, a) = γλe(s, a)

s = s′, a = a′

• Q(λ) algorithem:

Initialize: q(s, a) = 0, ∀(s, a), α ∈ (0, 1], ϵ > 0, denote ϵ-greedy policy

pi(a|s) by π loop (for each episode)

s = s_{0} e(s, a) = 0, ∀(s, a) get action a according to π based on q(s, a) while s is not terminal (for each step)

e(s, a) = e(s, a) + 1 take action a, get reward r and next state s′ get action a′ according to π based on q(s′, a) a^{∗} = arg max_{b} q(s′, b) δ = r + γq(s′, a^{∗}) − q(s, a) for all (s, a)

q(s, a) = q(s, a) + αδ e(s, a)

if a′ = a^{∗} then e(s, a) = γλe(s, a) else e(s, a) = 0 s = s′, a = a′

Note that Q(λ) algorithm is different from the SARSA(λ) algorithm in two ways. First, the action value is updated based on the greedy action a^{∗}, instead of the a′ by policy π; second, if the ϵ-greedy policy π happens to choose a random non-greedy action a′ ≠ a^{∗} with probability ϵ to explore rather than exploiting a^{∗}, the eligibility traces of all states are reset to zero. There are other different versions of the algorithm which do not reset these traces.

18.5 Value Function Approximation

All previoiusly considered algorithms are based on either the state values function v_{π}(s) for each state s or the action value q_{π}(s, a) for each state-action pair. As these function values can be considered as either a 1-D or 2-D table, the algorithms based on such function values are called tabular methods. However, when the number of states and the number of actions in each state are large (e.g., the game Go has 10^{170} states), the tabular methods are no longer suitable due to the unrealistic table sizes. In such a case, we can consider approximating the state value function v(s) or action-value function q(s, a) by a function v̂(s, w) or q̂(s, a, w), parameterized by a set of parameters as the components of vector w. By doing so the state or action value functions can be represented more conveniently in terms of a much smaller number of parameters in w.

This approach can be considered as a regression problem to fit a continuous function v̂(s, w) or q(s, a, w) to a set of discrete and finite data points v_{π}(s) or q_{π}(s, a) obtained by sampling the MDP of the environment, so that all points in the state space, including those for stataes or state-action pairs never actually observed during the sampling process can be represented.

Examples of such value function approximators include

• linear combination of all features

• multi-layer neural networks

• decision trees

Same as in regression problems, we desire to find the optimal parameter w for the modeling function ˆ(v)_{π}(s, w) so that the following objective function, the mean squared error between the approximated values and the sample values, is minimized:

J(w) = \frac{1}{2}E_{π}[(v_{π}(s) − v̂(s, w))^{2}]  (18.72)

where the expectation E_{π} is with respect to all states visited while following some policy π. The gradient vector of J(w) is

▽J(w) = \frac{d}{dw} J(w) = \frac{1}{2}E_{π} [\frac{d}{dw} [(v_{π}(s) − v̂(s, w))^{2}]]
= −E_{π} [(v_{π}(s) − v̂(s, w))▽v̂(s, w)]  (18.73)

and the optimal parameter vector w^{∗} can be found iteratively by the gradient descent method

w_{n+1} = w_{n} + Δw = w_{n} − α▽J(w) = w_{n} + αE_{π} [(v_{π}(s) − v̂(s, w))▽v̂(s, w)]

(18.74)

where α is the step size or learning rate, and

Δw = −α▽J(w) = αE_{π} [(v_{π}(s) − v̂(s, w))▽v̂(s, w)]  (18.75)

If the method of stochastic gradient descent (SGD) is used, i.e., the parameter w is updated based on only one data point at each iteration instead of the expectation of the data points, then E_{π} for the expectation can be dropped:

w_{n+1} = w_{n} + Δw = w_{n} + α(v_{π}(s) − v̂(s, w))▽v̂(s, w))  (18.76)

Here the true value function v_{π}(s), the expected return, is unknown and needs to be estimated by the actual return G_{t} at each state s = s_{t} based on the MC method, or r_{t+1} + γv̂_{π}(s′, w) in terms of the immediate reward r_{t+1} and the estimated value v̂_{π}(s′, w) of the next state s′ based on the TD method, by sampling the environment. Such an approach can be considered as a supervised learning based on labeled training data set: {(s_{t}, G_{t}), ∀t} for the MC method, or {(s_{t}, r_{t+1}, ∀t} for xthe TD method.

In the following, we will consider the special case where the state value v_{π}(s) is approximated by a linear function

v̂(s, w) = ∑_{i=1}^{d} x_{i}w_{i} = w^{T} x(s)  (18.77)

based on the assumption that the value function of each state can be approximated as a linear combinatioin of a set of features in the feature vector x(s) = [x_{1}(s), ⋯, x_{d}(s)]^{T} for state s, weighted by the corresponding weights in the weight vector w = [w_{1}, ⋯, w_{d}]^{T} for all states.

As a simple example, if we represent each of the N = |S| states s_{n} by a d = N dimensional binary feature vector x(s_{n}) = [x_{1}(s_{n}), ⋯, x_{N}(s_{n})]^{T} of all zero components x_{i}(s_{n}) = 0 except the n-th one x_{n}(s_{n}) = 1, then the weight vector can be w = [w_{1}, ⋯, w_{N}]^{T} = [v_{π}(s_{1}), ⋯, v_{π}(s_{N})]^{T}, and we have v̂_{π}(s_{n}, w) = w^{T} x(s_{n}) = v_{π}(s_{n}). In this case, each state s is represented by its value function v_{π}(s), same as in all previous discussions.

The objective function for the mean squared error of such a linear approximating function is

J(w) = \frac{1}{2}E_{π}[(v_{π}(s) − v̂(s, w))^{2}] = \frac{1}{2}E_{π}[(v_{π}(s) − w^{T} x(s))^{2}]  (18.78)

and its gradient is

▽J(w) = \frac{d}{dw} J(w) = −E_{π} [(v_{π}(s) − w^{T} x(s))x(s)]  (18.79)

If all data points in terms of state s represented by x(s) and the corresponding return G(s) as a sample of the true value v_{π}(s) have already been collected, then the value function approximation can be carried out as an off-line algorithm in a batch manner, the same as in the linear least squares linear regression problem discussed in a previous chapter, and the optimal weight vector that minimizes the squared error

w^{∗} = arg max_{x} ∑_{s}[G(s) − v̂_{π}(s, w)]^{2}  (18.80)

can be found by the same pseudo inverse method

w^{∗} = (XX^{T})^{−1}XG  (18.81)

where X = [x_{1}, ⋯, x_{N}] is a matrix containing all N samples for the states and G = [G_{1}, ⋯, G_{N}]^{T} is a vector containing the corresponding returns.

However, as reinforcement learning is typically an online problem, the parameter w for the approximator needs to be modified in real-time whenever a new piece of data becomes available during the sampling of the environment. In this case, we can find the gradient descent method, based on the gradient vector of J(w):

▽J(w) = \frac{d}{dw} J(w) = −E_{π}(v_{π}(s) − v̂(s, w))▽v̂(s, w)
= −E_{π}(v_{π}(s) − w^{T} x(s))x(s)  (18.82)

and the optimal parameter w^{∗} can be found iteratively

w_{n+1} = w_{n} + Δw = w_{n} + αE_{π} [v_{π}(s) − v̂(s, w)] ▽v̂(s, w)
= w_{n} + αE_{π} [v_{π}(s) − w^{T} x(s)] x(s)  (18.83)

where Δw is the increment in each iteration:

Δw = αE_{π} [v_{π}(s) − v̂(s, w)] ▽v̂(s, w) = αE_{π} [v_{π}(s) − w_{n}^{T} x(s)] x(s) (18.84)

If stochastic gradient descent is used, then w is updated whenever a new sample data point x(s_{t}) is available at a time step t, then the expectation E_{π} in the equations above can be dropped.

We note that the expectation of the estimated w found by this SGD method is the same as that obtained by full GD method. Also this iteration will convergence to the global minimum of the objective function J(w) in Eq. (18.78) as it is a quadratic function with only one minimum which is global.

Specifically, the state value function is unknown and can be estimated in several different ways, similar to the corresponding algorithms discussed before.

• MC method

The value function v_{π}(s) is estimated by the actual return G_{t} for each state s_{t} visited in an episode, obtained only at the end of each episode while sampling the environment:

Δw = α[G_{t} − v̂(s_{t}, w)]▽v̂(s_{t}, w) = α[G_{t} − w^{T} x(s_{t})]x(s_{t})  (18.85)

Here is the pseudo code for the algorithm based on the MC method, similar to that for value evaluation algorithms discussed previously: Initialize w = 0 loop (for each episode)

run current episode to the end to get G_{t}, t = 1, ⋯, T for t = 1, ⋯, T

if s_{t} is visited the first time

w = w + α[G_{t} − w^{T} x(s_{t})]x(s_{t})

end if end for

end loop

• TD(0) method

The value function v_{π}(s) is estimated by the TD target, the sum of the immediate reward r_{t+1} and the approximated value of the next state s_{t+1}, at each time step of each episode while sampling the environment:

Δw = α[r_{t+1} + γw^{T} x(s_{t+1}) − w^{T} x(s_{t})]x(s_{t})  (18.86)

Here is the pseudo code for the algorithm based on the TD method, similar to that for value evaluation algorithms discussed previously: Initialize w = 0 loop (for each episode)

for each time step in current episode take action a = π(s), get reward r and next state s′ w = w + α[r + γw^{T} x(s′) − w^{T} x(s)]x(s)

end for end loop

• TD(λ) method

The value function v_{π}(s) is estimated based on λ-return G_{t}^{λ} given in Eq. (18.62). As discussed before, TD(λ) has two flavors:

– Forward veiw: same as MC method, except the true return G_{t} is replaced by λ-return G_{t}^{λ} available only at the end of each spisode:

Δw = α[G_{t}^{λ} − w^{T} x(s_{t})]x(s_{t})  (18.87)

– Backward view: similar to TD(0), this is based on the immediate reward r_{t+1} available at every step of each episode:

δ_{t} = α[r_{t+1} + γw^{T} x(s_{t+1}) − w^{T} x(s_{t})]x(s_{t})  (18.88)
e_{t} = γλe_{t−1} + x(s_{t}), Δw = αδ_{t}e_{t}  (18.89)

Given a policy π for an MDP, we define a probability districution d(s) over all staes visited according to π, satisfying

∑_{s} d(s) = 1  (18.90)

and the following balance equation:

d(s′) = ∑_{s} ∑_{a} π(a|s)p(s′|s, a)d(s)  (18.91)

As both d(s′) and d(s) represent the same distribution, they must be identical.

Given the probability distribution d(s), the mean squared error of the value function approximation for a policy π can be expressed as

MSE(w) = ∑_{s} d(s)[v_{π}(s) − v̂_{π}(s, w)]^{2}  (18.92)

It can be shown that the MC method for the linear value function approximation will converge to the optimal weight vector w_{MC} that minimizes the mean squared error above:

MSE(w_{MC}) = min_{w} ∑_{s} d(s)[v_{π}(s) − v̂_{π}(s, w)]^{2}  (18.93)

18.6 Control based on Function Approximation

The control algorithms based on approximated value functions also follow the general method of the general policy iteration, as illustrated bellow:

We note that this is similar to the algorithms for model-free control illustrated in Fig. ??, but with the action-value function q_{π}(s, a) is replaced by the parameter w of the approximation action function q̂(s, a, w). In particular, for a linear function, we have:

q̂(s, a, w) = ∑_{n} w_{n}x(s, a) = w^{T} x_{n}(s, a)  (18.94)

where x(s, a) is the feature vector for the state-action pair (s, a). We need to find the optimal parameter w that minimizes the objective function, the mean square error of the approximation:

J(w) = \frac{1}{2}E_{π}[(q_{π}(s, a) − q̂(s, a, w))^{2}] = \frac{1}{2}E_{π}[(q_{π}(s, a) − w^{T} x(s, a))^{2}] (18.95)

with gradient vector:

▽J(w) = \frac{d}{dw} J(w) = −E_{π} [q_{π}(s, a) − q̂(s, a, w)] ▽q̂(s, a, w)
= −E_{π} [(q_{π}(s, a) − w^{T} x(s, a))x(s, a)]  (18.96)

If the stochastic gradient descent method is used based on a single sample of the action value q_{π}(s, a), instead of its expectation, then E_{π} can be dropped, and the optimal weight vector w^{∗} that minimizes J(w) in Eq. (18.95) can be learned iteratively:

w_{t+1} = w_{t} + Δw = w_{t} − α▽J(w)
= w_{t} + α [(q_{π}(s_{t}, a_{t}) − q̂_{π}(s_{t}, a_{t}, w))▽q̂_{π}(s_{t}, a_{t}, w)]
= w_{t} + α [(q_{π}(s_{t}, a_{t}) − w_{t}^{T} x(s_{t}, a_{t}))x(s_{t}, a_{t})]  (18.97)

where Δw is the increment of the update:

Δw = −α▽J(w) = α[q_{π}(s_{t}, a_{t}) − q̂_{π}(s_{t}, a_{t}, w)]▽q̂_{π}(s_{t}, a_{t}, w)
= α[q_{π}(s_{t}, a_{t}) − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})  (18.98)

As the true Q-value q_{π}(s, a) in the expression is unknown, it needs to be estimated by some target depending on the specific methods used:

• MC method:

The true q_{π}(s, a) is replaced by the sample return G_{t} as the target, obtained at the end of each episode:

Δw = α[G_{t} − q̂(s_{t}, a_{t}, w)]▽q̂(s_{t}, a_{t}, w) = α[G_{t} − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})

(18.99)

• TD(0) method:

The action value function q_{π}(s, a) is replaced by the TD target, the sum of the immediate reward, available at each step of each spisode, and the approximated action value of the next state s_{t+1}:

– SARSA (on-policy):

Δw = α[r_{t+1} + γq̂(s_{t+1}, a_{t+1}, w) − q̂(s_{t}, a_{t}, w)]▽q̂(s_{t}, a_{t}, w)
= α[r_{t+1} + γw^{T} x(s_{t+1}, a_{t+1}) − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})(18.100)

Following Eq. (18.71), the TD error is defined as:

δ_{t} = r_{t+1} + γq̂(s_{t+1}, a_{t+1}, w) − q̂(s_{t}, a_{t}, w)  (18.101)

then we get

Δw = αδ_{t}▽q̂(s_{t}, a_{t}, w)  (18.102)

– Q-learning (off-policy):

Δw = α[r_{t+1} + γ max_{a′} q̂(s_{t+1}, a′, w) − q̂(s_{t}, a_{t}, w)]▽q̂(s_{t}, a_{t}, w)
= α[r_{t+1} + γ max_{a′} w^{T} x(s_{t+1}, a′) − w_{t}^{T} x(s_{t}, a_{t})]x(s, a)(18.103)

• TD(λ) method:

In the forward-view version of the TD(λ) method the action function q_{π}(s, a) is approximated by λ-return G_{t}^{λ} as the target, available only at the end of each spisode:

Δw = α[G_{t}^{λ} − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})  (18.104)

The backward-view version of the TD(λ) method based on eligibility traces is more advantageous in both space and temporal complexity as well as learning efficiency.

We first define an eligiibility trace vector which is set to zero at the beginning of the episode, but then decays

e_{t} = γλe_{t−1} + ▽_{w}q̂(s_{t}, a_{t}, w)  (18.105)
Δw = αδ_{t}e_{t}  (18.106)

and Eq. (18.70)

v_{π}(s) = v_{π}(s) + αδ_{t}e_{t}(s),  ∀s ∈ S  (18.107)

/////

Recall the TD error for the backward view of the TD(λ) method first given in Eq. (18.71):

δ_{t} = (r_{t+1} + γv_{π}(s_{t+1}) − v_{π}(s))  (18.108)

In summary, here are the conceptual (not necessarily algorithmic) steps for the general model-free control based on approximated action-value function:

• Learn parameter w as in Eq. (18.97),

• Get the Q-values as in Eq. (18.94)

• Obtain the policy by ϵ-greedy approach as in Eq. (18.41)

These steps are also illustrated below:

Training of w ⟹ Q − value ⟹ Policy π  (18.109)

18.7 Deep Q-learning

Previously we approximated the state or action value functions by a linear function v̂_{π}(s, w) = w^{T} x(s), or q̂_{π}(s, a, w) = w^{T} x(s, a) parameterized by the weight vector w based on a set of features in x. However, these features need to hand picked or designed based on the specific problem to solve.

Network!

18.8 Policy Gradient Methods

All RL algorithms previous considered are based on either state or action-value function and the policy is indectly derived from them by greedy or ϵ-greedy method. However, as the ultimate goal of an RL problem is to find the optimal policy that maximizes the return, it make sense to consider this as an optimization problem to directly find the optimal policy based on an objective function representing the total return to be maximized, by the gradient ascent method, as discussed in the following.

Previously we approximate the state value function v_{π}(s) and action value function q_{π}(s, a) by some parameterized functions v̂_{π}(s, w) and q̂_{π}(s, a, w) respectively, of which the parameter w can be obtained by sampling the environment, as a supervised learning process. Now we construct a model of a stochastic policy as a parameterized function:

π(a|s, θ) = P (a|s, θ)  (18.110)

where vector θ = [θ_{1}, ⋯, θ_{d}]^{T} represents some d parameters of the model. As the dimensionality d is typically smaller than the number of states and actions, such a parameterized policy model is suitable in cases where the numbers of states and actions are large or even continuous.

As a specific example, the policy model can be based on the soft-max function:

π(a|s, θ) = \frac{e^{h(s,a,θ)}}{∑_{b} e^{h(s,b,θ)}}  (18.111)

satisfying ∑_{a} π(a|s, θ) = 1. Here the summation is over all possible actions, and h(s, a, θ) is the preference of action a in state s, which can be a parameterized function such as a simple linear function h(s, a, θ) = θ^{T} x(s, a), or a neural network with weights represented by θ, the same as how the value functions are approxmiated in Section 18.5. According to this policy, an action with higher preference h(s, a, θ) will have a higher probability to be chosen.

Different from how we find the parameter w of q̂(s, a, w) by minimizing J(w), the mean squared error between the value function q_{π}(s, a) and its model q̂_{π}(s, a, w), based on gradient descent, here we find the parameter θ of π(a|s, θ) by maximizing J(θ) representing the value function, the expected return, under the policy, based on gradient ascent. Such methods are therefore called policy gradient methods.

The value-function based methods considered previously and the policy-based methods considered here are summarized below, together with the actor-critic method, as the combination of the two:

• Value-based: the policy is derived by the greedy or ϵ-greedy method based on the value function learned during sampling the environment as a supervised learning process.

• Policy-based: the policy is directly learned without explicitly value functioin estimation.

• Actor-Critic: parameters for both the value function model and policy model are learned simultaneously.

Advantages of policy-based RL includes better convergence properties, but may stuck at local optimum, effective in high-dimensional or continuous action space can learn stochastic policies, but have high variance.

How good a policy is may be measured by different objective functions all related to the values or rewards associated with the policy being evaluated, depending on the environment of the specific problem:

• In episodic environments with finite horizon, we can use the value, the expected return (sum of discounted future rewards), of the start state s_{0}:

J_{1}(θ) = v_{π_{θ}}(s_{0})  (18.112)

• In continuing (online) environments with infinite horizon, we can use the average value

J_{avV} (θ) = ∑_{s} d_{π_{θ}}(s)v_{π_{θ}}(s)  (18.113)

where d_{π_{θ}}(s) is the stationary distribution of all states under policy π(a|s, θ).

We can find the optimal paramter θ^{∗} for the policy model π(a|s, θ) by solving the maximization problem:

θ^{∗} = arg max_{θ} J(θ)  (18.114)

based on the gradient of J(θ):

▽_{θ}J(θ) = \frac{d}{dθ} J(θ) = [\frac{∂J}{∂θ_{1}} , ⋯, \frac{∂J}{∂θ_{n}} ]^{T}  (18.115)

by the iterative gradient ascent method:

θ_{t+1} = θ_{t} + Δθ = θ_{t} + α▽_{θ}J(θ)  (18.116)

Here we use t for the index of the iteration based on the assumption that the a new sample point is available at every time step of an episode while sampling the environment following policy π(a|s, θ).

While it is conceptually straight forward to see how gradient ascent method can be used to find the optimal parameter θ^{∗} that maximizes J(θ), it is not easy to actually find its gradient ▽J(θ), which depends on not only what actions to take at the states directly determined by the policy π(a|s, θ), but also how the states are distributed under the policy in an unknown environment. This challenge is addressed by the following policy gradient theorem. The proof of the theorem below leads to an expression of the gradient ▽_{θ}J(θ), which can be used to find the optimal parameter θ^{∗} iteratively by the stochastic gradient ascend method.

Proof of policy gradient theorem:

▽_{θ}v_{π_{θ}}(s) = ▽_{θ} (∑_{a} π(a|s, θ)q_{π_{θ}}(s, a))
=^{1} ∑_{a} [▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a) + π(a|s, θ) ▽_{θ}q_{π_{θ}}(s, a)]
=^{2} ∑_{a} [▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a) + π(a|s, θ)▽_{θ} ∑_{s′} ∑_{r} P (s′, r|s, a)(r + v_{π_{θ}}(s′))]
=^{3} ∑_{a} [▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a) + π(a|s, θ) ∑_{s′} ∑_{r} P (s′, r|s, a)▽_{θ}v_{π_{θ}}(s′)]
=^{4} ∑_{a} [▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a) + π(a|s, θ) ∑_{s′} P (s′|s, a)▽_{θ}v_{π_{θ}}(s′)]
=^{5} ∑_{a} ▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a) + ∑_{a} ∑_{s′} π(a|s, θ)P (s′|s, a)▽_{θ}v_{π_{θ}}(s′)
=^{6} ϕ(s) + ∑_{s′} ∑_{a} P (s′|s, a)π(a|s, θ)▽_{θ}v_{π_{θ}}(s′)  (18.117)

where

1. product rule of derivative: (uv)′ = u′v + uv′

2. q_{π_{θ}}(s, a) = ∑_{s′} ∑_{r} P (s′, r|s, a)(r + v_{π_{θ}}(s′))

3. r is not a function of θ

4. ∑_{r} P (s′, r|s, a) = P (s′|s, a) in Eq. (18.17)

5. π(a|s, θ) is indpendent of s′ and moved inside summation over s′

6. We have defined

ϕ(s) = ∑_{a} ▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a)  (18.118)

We note that Eq. (refPolicyGradientThm1) is a recursion by which ▽_{θ}v_{π_{θ}}(s) is expressed as a function in terms ▽_{θ}v_{π_{θ}}(s′).

We further define P r_{π}(s → x, k) as the probability of transitioning from state s to a state x after k steps following π(a|s, θ):

s ⟶^{π(a|s,θ)} s′ ⟶^{π(a|s′,θ)} s′′ ⟶^{π(a|s′′,θ)} ⋯ ⟶^{π(a|s^{(k−1)},θ)} s^{(k)} = x  (18.119)

with the following properties:

•

P r_{π}(s → s, 0) = 1  (18.120)

•

P r_{π}(s → s′, 1) = ∑_{a} π(a|s, θ)P (s′|s, a)  (18.121)

•

∑_{s′} P r_{π}(s → s′, 1) = 1  (18.122)

•

P r_{π}(s → s′′, 2) = P r_{π}(s → s′, 1)P r_{π}(s′ → s′′, 1)  (18.123)

Continuing Eq. (18.117) we keep rolling out the recursion of ▽_{θ}v_{π_{θ}}(s′) and get:

▽_{θ}v_{π_{θ}}(s) = ϕ(s) + ∑_{s′} ∑_{a} π(a|s, θ)P (s′|s, a)▽_{θ}v_{π_{θ}}(s′)
=^{1} ϕ(s) + ∑_{s′} P r_{π}(s → s′, 1)▽_{θ}v_{π_{θ}}(s′)
=^{2} ϕ(s) + ∑_{s′} P r_{π}(s → s′, 1) [ϕ(s′) + ∑_{s′′} P r_{π}(s′ → s′′, 1)▽_{θ}v_{π_{θ}}(s′′)]
=^{3} ϕ(s) + ∑_{s′} P r_{π}(s → s′, 1)ϕ(s′) + ∑_{s′′} P r_{π}(s → s′, 1)P r_{π}(s′ → s′′, 1)▽_{θ}v_{π_{θ}}(s′′)
=^{4} ϕ(s) + ∑_{s′} P r_{π}(s → s′, 1)ϕ(s′) + ∑_{s′′} P r_{π}(s → s′′, 2)▽_{θ}v_{π_{θ}}(s′′)
=^{5} ϕ(s) + ∑_{s′} P r_{π}(s → s′, 1)ϕ(s′) + ∑_{s′′} P r_{π}(s → s′′, 2)ϕ(s′′) + ∑_{s′′′} P r_{π}(s → s′′′, 2)▽_{θ}v_{π_{θ}}(s′′′)
= · · · · · · =^{6} ∑_{x∈S} ∑_{k=0}^{∞} P r_{π}(s → x, k)ϕ(x)  (18.124)

where

1. Eq. (18.121)

2. unroll ▽_{θ}v_{π_{θ}}(s′)

3. Eq. (18.122)

4. Eq. (18.123)

5. unroll ▽_{θ}v_{π_{θ}}(s′′)

6. keep unrolling recursively to infinity, and Eq. (18.120)

Now the gradient of the objective J(θ) = v_{π_{θ}}(s_{0}) can be written as

▽_{θ}J(θ) = ▽_{θ}v_{π}(s_{0}) = ∑_{s∈S} ∑_{k=0}^{∞} P r_{π}(s_{0} → s, k) ϕ(s)
= ∑_{s∈S} η(s) ϕ(s) = (∑_{s′} η(s′)) ∑_{s∈S} \frac{η(s)}{∑_{s′} η(s′)} ϕ(s)
∝ ∑_{s∈S} \frac{η(s)}{∑_{s′} η(s′)} ϕ(s) = ∑_{s∈S} μ_{π}(s) ∑_{a} ▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a)(18.125)

Here the proportionality is introduced due to the dropping of ∑_{s′} η(s′) as a constant independent of state s, and we also defined

η(s) = ∑_{k=0}^{∞} P r_{π}(s_{0} → s, k),  (18.126)

as the sum of all probabilities for visiting state s from the start state s_{0}, and

μ_{π}(s) = \frac{η(s)}{∑_{s′} η(s′)}  (18.127)

as a normalized verion of η(s) representing the probability distribution of visiting state s while following policy π. Eq. (18.125) can be further written as

▽_{θ}J(θ) = ▽_{θ}v_{π}(s_{0}) ∝ ∑_{s∈S} μ_{π}(s) ∑_{a} π(a|s, θ)\frac{▽_{θ}π(a|s, θ)}{π(a|s, θ)} q_{π_{θ}}(s, a)
= ∑_{s∈S} μ_{π}(s) ∑_{a} π(a|s, θ) ▽_{θ} ln π(a|s, θ) q_{π_{θ}}(s, a)
= E_{π_{θ}} [▽_{θ} ln π(a|s, θ) q_{π_{θ}}(s, a)]  (18.128)

where E_{π_{θ}} denotes the expectation over all actions in each state s weighted by π(a|s, θ) and all states s ∈ S weighted by μ_{π}(s).

Q.E.D.

We see that the gradient ▽_{θ}J(θ) is now expressed as the expectation of the action value function q_{π}(s, a), weighted by the gradient of the logrithm of the corresponding policy π(a|s, θ). Now Eq. (18.116) can be further written as

θ_{t+1} = θ_{t} + α▽_{θ}J(θ) = θ_{t} + αE_{π_{θ}} [q_{π_{θ}}(s_{t}, a_{t}) ▽_{θ} ln π(a_{t}|s_{t}, θ)]  (18.129)

We further note that Eq. (18.125) still hold if an arbitrary bias term b(s) independent of action a is included:

▽_{θ}J(θ) ∝ ∑_{s∈S} μ_{π}(s) ∑_{a} ▽_{θ}π(a|s, θ) (q_{π_{θ}}(s, a))
= ∑_{s∈S} μ_{π}(s) ∑_{a} ▽_{θ}π(a|s, θ) (q_{π_{θ}}(s, a) + b(s))  (18.130)

as

∑_{a} ▽_{θ}π(a|s, θ) b(s) = b(s)▽_{θ} ∑_{a} π(a|s, θ) = b(s)▽_{θ}1 = 0  (18.131)

Now Eq. (18.116) can also be written as

θ_{t+1} = θ_{t} + α▽_{θ}J(θ) = θ_{t} + αE_{π_{θ}} [(q_{π_{θ}}(s_{t}, a_{t}) + b(s_{t})) ▽_{θ} ln π(a_{t}|s_{t}, θ)] (18.132)

The expectation E_{π_{θ}} in Eqs. (18.129) and (18.132) can be dropped if the method of stochastic gradient ascent is used, as in all algorithms below, where each iterative step is based on only one sample data point instead of its expectation.

Specifically, for a soft-max policy model as given in Eq. (18.111), we have

▽_{θ} ln π(a|s, θ) = ▽_{θ} ln (\frac{e^{θ^{T} x(s,a)}}{∑_{b} e^{θ^{T} x(s,b)}} ) = ▽_{θ} (θ^{T} x(s, a) − ln ∑_{b} e^{θ^{T} x(s,b)})

= x(s, a) − \frac{▽_{θ} ∑_{b} e^{θ^{T} x(s,b)}}{∑_{b} e^{θ^{T} x(s,b)}} = x(s, a) − \frac{∑_{b} e^{θ^{T} x(s,b)}x(s, b)}{∑_{b} e^{θ^{T} x(s,b)}}
= x(s, a) − ∑_{b} \frac{e^{θ^{T} x(s,b)}}{∑_{b} e^{θ^{T} x(s,b)}} x(s, b)
= x(s, a) − ∑_{b} π(b|s, θ)x(s, b)  (18.133)

We list set of popular policy-based algorithms below, based on either the MC or TD methods, generally used in previous algorithms.

• REINFORCE (MC) policy gradient

In this algorithm, the action value function q_{π_{θ}}(s_{t}, a_{t}) in Eq. (18.129) is replaced by the return G_{t} btained at the end of each episode while sampling the environment. Now the equation becomes:

θ_{t+1} = θ_{t} + αγ^{t}G_{t} ▽_{θ} ln π(a|s, θ)  (18.134)

Note that the discount factor γ^{t} is included as the expression for ▽_{θ}J(θ) in Eq. (18.128) assumed γ = 1 for simplicity.

Here is the pseudo code for the algorithm: Initialize π(a|s, θ loop (for each episode)

At the end of episode, get G_{t} for each state visited for t = 1, ⋯, T

θ = θ + αγ^{t}G_{t}▽_{θ} ln π(a_{t}|s_{t}, θ)

end loop

• Actor-Critic (TD) policy gradient

As its name suggests, this algorithm is based on two approximation function models, the first for the policy π(a|s, θ) parameterized by θ, the actor, same as in the REINFORCE algorithm above, and the second for the value function v̂_{π}(s, w) parameterized by w, the critic, same as in Eq. (18.97).

Specifically, in Eq. (18.132), the action value function q_{π_{θ}}(s_{t}, a_{t}) is replaced by its bootstrapping expression r_{t+1} +γv̂_{π}(s′, w), and the bias term b(s) is replaced by the approximated value function v̂(s_{t}, w). Now both parameters w and θ can be found iteratively at every step of an episode while sampling the environment (the TD method) by stochastic gradient method with the expectation E_{π} dropped:

w_{t+1} = w_{t} + α_{w} [(r_{t+1} + γv̂_{π}(s′, w) − v̂_{π}(s, w)) ▽_{w}v̂_{π}(s, w)]
θ_{t+1} = θ_{t} + α_{θ} [(r_{t+1} + γv̂_{π}(s′, w) − v̂_{π}(s, w)) ▽_{θ} ln π(a|s, θ)] (18.135)

Here is the pseudo code for the algorithm: initialize model parameters θ and w initialize step sizes α_{θ} and α_{w} loop (for each episode)

initialize s = s_{0}, t = 0 while s is not terminal (for each step)

take action a following π(a|s, θ), find reward r and next

state s′ find TD error: δ = r + γv̂(s′, w) − v̂(s, w) w = w + α_{w} δ ▽v̂(s, w) θ = θ + α_{θ}γ^{t}δ ▽ ln π(a|s, θ) s = s′ t = t + 1

end while end loop

• Backward view of TD(λ) policy gradient

Here is the pseudo code for the algorithm: initialize model parameters θ and w initialize step sizes α_{θ} and α_{w} initialize trace-decay rates λ_{θ} and λ_{w} loop (for each episode)

initialize s = s_{0}, t = 0 initialize z_{θ} = 0, z_{w} = 0 while s is not terminal (for each step)

take action a following π(a|s, θ), find reward r and next

state s′ find TD error: δ = r + γv̂(s′, w) − v̂(s, w) z_{θ} = γλ_{θ}z_{θ} + ▽γ^{t} ln π(a|s, θ) z_{w} = γλ_{w}z_{w} + γ^{t}▽v̂(s, w) w = w + α_{w} δ ▽v̂(s, w) θ = θ + α_{θ} δ ▽ ln π(a|s, θ) s = s′ t = t + 1

end while end loop

Following the similar steps, the policy gradient theorem for environment with continuous state and action spaces can be also proven:

▽_{θ}J(θ) = ∫_{s} μ_{π}(s) ∫_{a} ▽_{θ}π(a|s, θ) q_{π_{θ}}(s, a)da ds  (18.136)

Appendix A A Review of Linear Algebra

A.1 Inner Product Space

• Vector space

A vector space over field F is a set V which is closed under the following two operations of addition and scalar multiplication defined for its members (called vectors), i.e., the results of the operations are also members of V. 1.The vector addition that maps any two vectors x, y ∈ V to another vector x + y ∈ V satisfying the following properties:

– Commutativity: x + y = y + x.

– Associativity: x + (y + z) = (x + y) + z.

– Existence of zero: there is a vector 0 ∈ V such that: 0+x = x+0 = x.

– Existence of inverse: for any vector x ∈ V, there is another vector −x ∈ V such that x + (−x) = 0.

2.The scalar multiplication that maps a vector x ∈ V and scalar a ∈ F (F can be a real or complex field) to another vector ax ∈ V with the following properties:

– a(x + y) = ax + ay.

– (a + b)x = ax + bx.

– abx = a(bx).

– 1x = x. where 0 = [0, ⋯, 0]^{T} and 1 = [1, ⋯, 1]^{T} are two constant vectors.

A subset W of V is a subspace of V, denoted by W ⊆ V, if it is also a vector space, i.e., it is closed under the same operations defined in V: 1.The zero vector 0 of V must be in W (the zero vector is unique in V, which W must have). 2.For any x, y ∈ W, x + y ∈ W. 3.For any x ∈ W, ax ∈ W.

Listed below is a set of typical vector spaces:

– n-D vector space ℝ^{n} or ℂ^{n} This space contains all n-D vectors expressed as an n-tuple, an ordered list of n elements (or components):

x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] = [x_{1}, …, x_{n}]^{T} ,  (A.1)

which can be used to represent a discrete signal containing n samples. We will always represent a vector as a vertical or column vector, or the transpose of a horizontal or row vector. The space is denoted by either ℂ^{n} if the elements are complex x_{i} ∈ ℂ, or ℝ^{n} if they are all real x_{i} ∈ ℝ (i = 1, …, n). A subspace of ℝ^{n} can be a ℝ^{m} (m < n) that passes origin (zero). For example, any 2-D plane passing through the origin of a 3-D space is its subspace. However, if the 2-D plane does not pass through the origin, it is not a subspace. Also, as 3-D cube or sphere centered at the origin is not a subspace, as it is not closed under the operations of addition and scalar multiplication.

– A vector space can be defined to contain all m × n matrices composed of n m-D column vectors:

A = [a_{1}, …, a_{n}] = [ \begin{matrix}a_{11} & a_{12} & ⋯ & x_{1n} \\ x_{21} & x_{22} & ⋯ & x_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ x_{m1} & x_{m2} & ⋯ & x_{mn} \end{matrix} ] ,  (A.2)

where the ith column is an m-D vector a_{i} = [x_{1i}, …, x_{mi}]^{T}. Such a matrix can be converted to an mn-D vector by cascading all of the column (or row) vectors.

– l^{2} space:

The concept of an n-D space ℝ^{n} or ℂ^{n} can be generalized by allowing the dimension n to become to infinity so that a vector in the space becomes a sequence x = […, x_{i}, …]^{T} for 0 ≤ i < ∞ or −∞ < i < ∞. If all vectors are square-summable, the space is denoted by l^{2}. All discrete energy signals are vectors in l^{2}.

– L^{2} space:

A vector space can also be a set of real or complex valued continuous functions x(t) defined over either a finite range such as 0 ≤ t < T, or an infinite range −∞ < t < ∞. If all functions are square-integrable, the space is denoted by L^{2}. All continuous energy signals are vectors in L^{2}.

Note that the term “vector”, generally denoted by x in the following, may be interpreted in two different ways. First, in the most general sense, it represents a member of a vector space, such as any of the vector spaces considered above; e.g., a function x = x(t) ∈ L^{2}. Second, in a more narrow sense, it can also represent a tuple of n elements, an n-D vector x = [x_{1}, …, x_{n}]^{T} ∈ ℂ^{n}, where n may be infinity. It should be clear what a vector x represents from the context.

• Linear independence

A set of vectors {v_{1}, ⋯, v_{n}} are linearly independent if none of them can be represented as a linear combination of the others.

Theorem: For any set of linearly independent vectors {v_{1}, ⋯, v_{n}}, if their linear combination is zero

∑_{j=1}^{n} c_{j}v_{j} = 0  (A.3)

then all coefficients must be zero c_{1} = ⋯ = c_{n} = 0, or c = 0.

Proof: Assume c_{k} ≠ 0, then we would get

v_{k} = −\frac{1}{c_{k}} ∑_{j=1, i≠k}^{n} c_{j}v_{j}  (A.4)

i.e., v_{k} is a linear combination of the remaining n − 1 vectors, in contradiction with the assumption that they are independent.

In particular, in the n-D Euclidean space, the n vectors can be written as v_{j} = [v_{1j}, ⋯, v_{nj}]^{T} (j = 1, ⋯, n), and their linear combination

∑_{j=1}^{n} c_{j}v_{j} = [v_{1}, ⋯, v_{n}] [ c_{1} ; ⋮ ; c_{n} ] = [ \begin{matrix}c_{11} & ⋯ & c_{1n} \\ ⋮ & ... & ⋮ \\ c_{n1} & ⋯ & c_{nn} \end{matrix} ] [ c_{1} ; ⋮ ; c_{n} ] = Vc

(A.5)

For this to be a zero vector, i.e., for the homogeneous equation Vc = 0 to hold, the coefficient vector c has to be zero, as V is a full rank matrix. QED

• Convex combination

The convex combination of n vectors x_{1}, ⋯, x_{n} is their sum weighted by coefficients {c_{1}, ⋯, c_{n}} that add up to 1:

∑_{i=1}^{n} c_{i}x_{i},  ∑_{i=1}^{n} c_{i} = 1  (A.6)

The convex hull of these points is the set of all their convex combinations. For example, the convex hull of three points in a plane is the triangle formed by these points as vertices, in which any point is a convex combination of the three vertices.

• Inner product

An inner product in a vector space V is a function that maps two vectors x, y ∈ V to a scalar ⟨x, y⟩ ∈ ℂ or ℝ and satisfies the following conditions:

– Positive definiteness:

⟨x, x⟩ ≥ 0,  ⟨x, x⟩ = 0 iff x = 0.  (A.7)

– Conjugate symmetry:

⟨x, y⟩ = \overline{⟨y, x⟩}.  (A.8)

If the vector space is real, the inner product becomes symmetric:

⟨x, y⟩ = ⟨y, x⟩.  (A.9)

– Linearity in the first variable:

⟨ax + by, z⟩ = a⟨x, z⟩ + b⟨y, z⟩,  (A.10)

where a, b ∈ ℂ. The linearity does not apply to the second variable unless the coefficients are real a, b ∈ ℝ:

⟨x, ay + bz⟩ = \overline{⟨ay + bz, x⟩} = \overline{a⟨y, x⟩ + b⟨z, x⟩} = \overline{a}⟨x, y⟩ + \overline{b}⟨x, z⟩

≠ a⟨x, y⟩ + b⟨x, z⟩

In the special case when b = 0, we have

⟨ax, y⟩ = a⟨x, y⟩, ⟨x, ay⟩ = \overline{a}⟨x, y⟩.  (A.11)

More generally we have

⟨∑_{n} c_{n}x_{n}, y⟩ = ∑_{n} c_{n}⟨x_{n}, y⟩, ⟨x, ∑_{n} c_{n}y_{n}⟩ = ∑_{n} \overline{c}_{n}⟨x, y_{n}⟩.

(A.12)

An inner product space is a vector space with inner product defined. In particular, when the inner product is defined, ℂ^{n} is called a unitary space and ℝ^{n} is called a Euclidean space.

Some examples of the inner product are listed below:

– In an n-D vector space, the inner product, also called the dot product, of two vectors x = [x_{1}, …, x_{n}]^{T} and y = [y_{1}, …, y_{n}]^{T} is defined as

⟨x, y⟩ = x^{T} \overline{y} = y^{∗}x = [x_{1}, …, x_{n}] [ \overline{y}_{1} ; ⋮ ; \overline{y}_{n} ] = ∑_{i=1}^{n} x_{i}\overline{y}_{i}, (A.13)

where y^{∗} = \overline{y}^{T} is the conjugate transpose of y.

– In a space of 2-D matrices containing m × n elements, the inner product of two matrices A and B is defined as

⟨A, B⟩ = ∑_{i=1}^{m} ∑_{j=1}^{n} a_{ij}\overline{b}_{ij}.  (A.14)

When the column (or row) vectors of A and B are concatenated to form two mn-D vectors, their inner product takes the same form as that of two n-D vectors.

– In a function space, the inner product of two function vectors x = x(t) and y = y(t) is defined as

⟨x(t), y(t)⟩ = ∫_{a}^{b} x(t)\overline{y(t)} dt = \overline{∫_{a}^{b} y(t)\overline{x(t)}} dt = \overline{⟨y(t), x(t)⟩}. (A.15)

– The covariance of two random variables x and y can be considered as an inner product

⟨x, y⟩ = σ_{xy}^{2} = E[(x − μ_{x})\overline{(y − μ_{y})}] = E(x\overline{y}) − μ_{x}\overline{μ}_{y}  (A.16)

Specially when μ_{x} = μ_{x} = 0, we have

⟨x, y⟩ = E[x\overline{y}].  (A.17)

• Vector norm

In general, the norm ||x|| of a vector x ∈ V is a non-negative real scalar that measures its size or length. ||x|| = 0 if and only if x = 0. There exist different definitions for vector norm, as shown in Section ??. A vector x is normalized (becomes a unit vector) if ||x|| = 1. Any given vector can be normalized when divided by its own norm: x/||x||.

The most widely used 2-norm of a vector is defined as

||x|| = \sqrt{⟨x, x⟩} = ⟨x, x⟩^{1/2},  (A.18)

The vector norm squared ||x||^{2} = ⟨x, x⟩ can be considered as the energy of the vector. In particular, in an n-D unitary space, the 2-norm of a vector x = [x_{1}, …, x_{n}]^{T} ∈ ℂ^{n} is:

||x|| = \sqrt{⟨x, x⟩} = \sqrt{x^{T} \overline{x}} = \sqrt{x^{∗}x} = (∑_{i=1}^{n} x_{i}\overline{x}_{i})^{1/2} = (∑_{i=1}^{n} |x_{i}|^{2})^{1/2} .

(A.19)

The total energy contained in this vector is its norm squared:

E = ||x||^{2} = ⟨x, x⟩ = ∑_{i=1}^{n} |x_{i}|^{2}.  (A.20)

Similarly, in a function space, the norm of a function vector x = x(t) is defined as

||x|| = (∫_{a}^{b} x(t)\overline{x(t)} dt)^{1/2} = (∫_{a}^{b} |x(t)|^{2} dt)^{1/2} ,  (A.21)

where the lower and upper integral limits a < b are two real numbers, which may be extended to all real values ℝ in the entire real axis −∞ < t < ∞. This norm exists only if the integral converges to a finite value; i.e., x(t) is an energy signal containing finite energy:

∫_{−∞}^{∞} |x(t)|^{2} dt < ∞.  (A.22)

All such functions x(t) satisfying the above are square-integrable, and they form a function space denoted by L^{2}(ℝ).

• Cauchy-Schwarz inequality

The Cauchy-Schwarz inequality holds for any two vectors x and y in an inner product space V:

|⟨x, y⟩|^{2} ≤ ⟨x, x⟩⟨ y, y⟩; i.e., 0 ≤ |⟨x, y⟩| ≤ ||x|| ||y||. (A.23)

Proof: If either x or y is zero, ⟨x, y⟩ = 0, the theorem holds (an equality). Otherwise, we consider the following inner product:

⟨x − λy, x − λy⟩ = ||x||^{2} − \overline{λ}⟨x, y⟩ − λ⟨y, x⟩ + |λ|^{2}||y||^{2} ≥ 0, (A.24)

where λ ∈ ℂ is an arbitrary complex number, which can be assumed to be:

λ = \frac{⟨x, y⟩}{||y||^{2}}, then \overline{λ} = \frac{⟨y, x⟩}{||y||^{2}}, |λ|^{2} = \frac{|⟨x, y⟩|^{2}}{||y||^{4}} .  (A.25)

Substituting these into the previous equation we get

||x||^{2} − \frac{|⟨x, y⟩|^{2}}{||y||^{2}} ≥ 0; i.e., |⟨x, y⟩| ≤ ||x|| ||y||.  (A.26)

The equation holds only if x − λy = 0 or x = λy, i.e., the two vectors are linearly dependent.

• Distance between two vectors

The distance d(x, y) between two vectors x and y is a real constant that satisfies:

– d(x, y) = 0 iff x = y

– d(x, y) = d(y, x)

– d(x, y) ≤ d(x, z) + d(z, y) In an inner-product space in which the inner product ⟨x, y⟩ between any two vectors x and y is defined, the distance between the two points can be defined as the norm of the difference between the two points:

d(x, y) = ||x − y|| = \sqrt{⟨(x − y), (x − y)⟩}  (A.27)

The norm of a vector can now be seen as its distance to the origin 0 of the space: ||x|| = ||x − 0||.

A vector space V is a metric space if a distance (or metric) d(x, y) between any two vector (two points) x and y is defined.

• Angle between two vectors The angle between two vectors x and y is defined as

θ = cos^{−1} (\frac{⟨x, y⟩}{||x|| ||y||}) .  (A.28)

Now the inner product of x and y can also be written as

⟨x, y⟩ = ||x|| ||y|| cos θ ≤ ||x|| ||y||  (A.29)

This is the Cauchy-Schwarz inequality. In particular, – If θ = 0, cos θ = 1, then x and y are collinear or linearly dependent, and

the inner product is maximized:

⟨x, y⟩ = ||x|| ||y||  (A.30)

i.e., the Cauchy-Schwarz inequality becomes an equality.

– If 0 < θ < π/2, 0 < cos θ < 1, we get the Cauchy-Schwarz inequality:

⟨x, y⟩ < ||x|| ||y||  (A.31)

– If θ = π/2, cos θ = 0, then x and y are orthogonal to each other, and the inner product is minimized:

⟨x, y⟩ = 0  (A.32)

Two vectors x and y are orthogonal or perpendicular to each other, denoted by x ⊥ y, if their inner product is zero ⟨x, y⟩ = 0, i.e., the angle between them is θ = cos^{−1} 0 = π/2.

• Projection

The orthogonal projection of a vector x ∈ V onto another vector y ∈ V is defined as a vector

p_{y}(x) = \frac{⟨x, y⟩}{||y||} \frac{y}{||y||} = \frac{⟨x, y⟩}{⟨y, y⟩}y = ||x|| cos θ \frac{y}{||y||},  (A.33)

where y/||y|| is the unit vector along the direction of y. In particular, if y is normalized with ||y|| = 1, then

p_{y}(x) = ⟨x, y⟩ y = ||x|| cos θ y  (A.34)

Note that

||p_{y}(x)|| = ||x|| cos θ  (A.35)

If only the magnitude of the projection is of interest, the unit vector y/||y|| can be dropped:

p_{y}(x) = ⟨x, y⟩/||y||  (A.36)

• Cauchy space

A sequence of points in a metric space x_{0}, x_{1}, x_{2}, ⋯ is a Cauchy sequence if it converges, i.e., for any ϵ > 0, there exists an integer N > 0 so that the following is true for any m, n > N:

d(x_{m}, x_{n}) < ϵ  (A.37)

If the limit of any Cauchy sequence of points in the space is also in the space, the space is complete, referred to as a Cauchy space.

A.2 Orthogonal Basis and Gram-Schmidt Process

Assume the n-D vector space ℝ^{n} or ℂ^{n} is spanned by a set of n independent basis vectors {v_{1}, ⋯, v_{n}}, not necessarily orthogonal, so that any vector x ∈ ℝ^{n} can be represented as a linear combination of the basis vectors:

x = ∑_{i=1}^{n} c_{i}v_{i} = [ v_{1} ⋯ v_{n} ] [ c_{1} ; ⋮ ; c_{n} ] = Vc  (A.38)

where V = [v_{1}, ⋯, v_{n}] is an n × n matrix composed of the n vectors and the coefficients in c = [c_{1}, ⋯, c_{n}]^{T} can be found by solving the linear equation system to get c = V^{−1}x with complexity is O(n^{3}).

These linearly independent vectors v_{1}, ⋯, v_{n} can be converted into a set of orthogonal vectors {u_{1}, ⋯, u_{n}} satisfying u_{i}^{T} u_{j} = 0 (i ≠ j) by the following Gram-Schmidt process:

u_{1} = v_{1}
u_{2} = v_{2} − p_{u_{1}}(v_{2}) = v_{2} − (\frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1}
u_{3} = v_{3} − p_{u_{1}}(v_{3}) − p_{u_{2}}(v_{3}) = v_{3} − (\frac{v_{3}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1} − (\frac{v_{3}^{T} u_{2}}{u_{2}^{T} u_{2}} ) u_{2}
· · · · · · · · ·
u_{k} = v_{k} − ∑_{i=1}^{k−1} p_{u_{i}}(v_{k}) = v_{k} − ∑_{i=1}^{k−1} (\frac{v_{k}^{T} u_{i}}{u_{i}^{T} u_{i}} ) u_{i}  (A.39)

where p_{u_{i}}(v_{k}) is the projection of v_{k} onto u_{i}:

p_{u_{i}}(v_{k}) = (\frac{v_{k}^{T} u_{i}}{u_{i}^{T} u_{i}} ) u_{i}  (A.40)

We see that u_{2} so obtained is indeed orthogonal to all u_{1}

u_{2}^{T} u_{1} = [v_{2} − (\frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1}]^{T} u_{1} = v_{2}^{T} u_{1} − \frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} u_{1}^{T} u_{1} = 0  (A.41)

and, by mathematical induction, we can further prove that u_{k}^{T} u_{i} = 0 for all i = 1, ⋯, k − 1. In other words, u_{k} is the component of v_{k} that is orthogonal to all previously found orthogonal vectors u_{1}, ⋯, u_{k−1}.

Given a set of orthogonal basis {u_{1}, ⋯, u_{n}} satisfying ⟨u_{i}, u_{j}⟩ = u_{i}^{T} u_{j} = 0 for any i ≠ j, we can again represent any given vector x as a linear combination of these basis as

x = ∑_{i=1}^{n} d_{i}u_{i}  (A.42)

The coefficients d_{i} can now be obtained easily. Premultiplying u_{j}^{T} on both sides of the equation above, we get

u_{j}^{T} x = u_{j}^{T} (∑_{i=1}^{n} d_{i}u_{i}) = ∑_{i=1}^{n} d_{i}u_{j}^{T} u_{i} = d_{j}u_{j}^{T} u_{j}  (A.43)

Solving for c_{j} we get

d_{j} = \frac{u_{j}^{T} x}{u_{j}^{T} u_{j}} , j = 1, ⋯, n  (A.44)

As the n coefficients are decoupled, each of them can be obtained separately with linear computational complexity O(n) with total complexity O(n^{2}) for all n of them. The complexity is reduced to O(n^{2}) from O(n^{3}) for solving the equation system c = V^{−1}b, needed in the case where the basis is not orthogonal. This is the reason why orthogonal bases are preferred in general. Now the vector x can be written as

x = ∑_{i=1}^{n} d_{i}u_{i} = ∑_{i=1}^{n} (\frac{u_{j}^{T} x}{u_{j}^{T} u_{j}} ) u_{i} = ∑_{i=1}^{n} p_{u_{i}}(x)  (A.45)

where the ith term of the summation above is simply the projection p_{u_{i}}(x) of x onto the ith basis vector u_{i}.

The concept of orthogonal vectors satisfying u^{T} v = 0 can be generalized to conjugate vectors that satisfy u^{T} Av = 0 with respect to a symmetric matrix A = A^{T}. This can be expressed in the form of inner product

u^{T} Av = (Av)^{T} u = v^{T} Au  (A.46)

Conjugate vectors with respect to A can also be considered as orthogonal to each other with respect to A. Two orthogonal vectors satisfying u^{T} v = 0 can be considered as a special case of conjugate vectors with respect to A = I.

Based on this generalized orthogonality, we can also define the projection of v onto u with respect to A as:

p_{u}(v) = (\frac{u^{T} Av}{u^{T} Au}) u  (A.47)

Given a set of A-conjugate basis vectors {d_{1}, ⋯, d_{n}} satisfying d_{i}^{T} Ad_{j} = 0 for any i ≠ j, we can represent any x as

x = ∑_{i=1}^{n} a_{i}d_{i}  (A.48)

The coefficients a_{i} can be obtained by pre-multiplying d_{j}^{T} A on both sides:

d_{j}^{T} Ax = ∑_{i=1}^{n} a_{i}d_{j}^{T} Ad_{i} = a_{j}d_{j}^{T} Ad_{j}  (A.49)

Solving for a_{j} we get

a_{j} = \frac{d_{j}^{T} Ax}{d_{j}^{T} Ad_{j}} , j = 1, ⋯, n  (A.50)

and x is now represented as the sum of its A-projections onto the n basis vectors {d_{1}, ⋯, d_{n}}:

x = ∑_{i=1}^{n} a_{i}d_{i} = ∑_{i=1}^{n} (\frac{d_{i}^{T} Ax}{d_{i}^{T} Ad_{i}} ) d_{i} = ∑_{i=1}^{n} p_{d_{i}}(x)  (A.51)

A set of independent basis vectors {v_{1}, ⋯, v_{n}} can now also converted to an orthogonal basis with respect to a symmetric vector A by Gram-Schmidt process:

d_{k} = v_{k} − ∑_{i=1}^{k−1} p_{d_{i}}(v_{k}) = v_{k} − ∑_{i=1}^{k−1} (\frac{v_{k}^{T} Ad_{i}}{d_{i}^{T} Ad_{i}} ) d_{i},  (k = 1, ⋯, n) (A.52)

A.3 Rank, Trace, Determinant, Transpose, and Inverse of Matrices

Let A be an m × n square matrix:

A = [a_{1}, ⋯, a_{n}] = [ \begin{matrix}a_{11} & ⋯ & a_{1n} \\ ⋮ & ... & ⋮ \\ a_{m1} & ⋯ & a_{mn} \end{matrix} ]_{m×n}  (A.53)

where

a_{j} = [ a_{1j} ; ⋮ ; a_{mj} ] ,  (j = 1, ⋯, n)  (A.54)

is the jth column vector and [a_{i1} ⋯ a_{in}] is the ith row vector (i = 1, ⋯, m). If m = n, A is a square matrix. In particular, if all entries of a square matrix are zero except those along the diagonal, it is a diagonal matrix. Moreover, if the diagonal entries of a diagonal matrix are all one, it is the identity matrix:

I = [ \begin{matrix}1 & 0 & ⋯ & 0 \\ 0 & 1 & & ⋮ \\ ⋮ & & ... & 0 \\ 0 & ⋯ & 0 & 1 \end{matrix} ]  (A.55)

• Rank

The m row vectors span the row space of A and the n columns vectors span the column space of A. The rank of each space is its dimension, the number of independent vectors in the space. The row and column spaces have the same rank, which is also the rank of matrix A, i.e.:

r = rank(A) ≤ min(m, n)  (A.56)

In other words, the rank of matrix A is the number of its independent rows or columns.

• Transpose The transpose x^{T} a column vector x is a row vector:

x^{T} = [ x_{1} ; ⋮ ; x_{n} ]^{T} = [x_{1}, ⋯, x_{n}]  (A.57)

The transpose A^{T} of a matrix A is obtained by switching the positions of elements a_{ij} and a_{ji} for all i ∈ {1, ⋯, m} and j ∈ {1, ⋯, n}. In other words, the ith column of A becomes the ith row of A^{T}, or equivalently, the ith row of A becomes the ith column of A^{T}:

A^{T} = [a_{1} ⋯ a_{n}]^{T} = [ a_{1}^{T} ; ⋮ ; a_{n}^{T} ]  (A.58)

where vector a_{i} is the ith column of A and its transpose a_{i}^{T} is the ith row of A^{T}.

The properties of the transpose:

– (A^{T})^{T} = A

– (A + B)^{T} = A^{T} + B^{T}

– (AB)^{T} = B^{T} A^{T}

– A^{T} and A have the same eigenvalues and eigenvectors.

If A = A^{T}, it is a symmetric matrix.

• Conjugate transpose

The conjugate transpose of a matrix A, denoted by A^{∗}, is by taking the complex conjugate of the its transpose:

A^{∗} = (\overline{A})^{T} = \overline{A^{T}}  (A.59)

i.e., (A^{∗})_{ij} = \overline{A}_{ji}, the ij-th entry of A^{∗} is the complex conjugate of the ji-th entry of A.

The properties of the conjugate transpose:

– (A^{∗})^{∗} = A

– (cA)^{∗} = \overline{c}A^{∗}

– (A + B)^{∗} = A^{∗} + B^{∗}

– (AB)^{∗} = B^{∗}A^{∗}

– det(A^{∗}) = (det A)^{∗} = \overline{det A}

– tr(A^{∗}) = (trA)^{∗} = \overline{trA}

– (A^{∗})^{−1} = (A^{−1})^{∗}.

– The eigenvalues of A^{∗} are the complex conjugates of the eigenvalues of A. (But their eigenvectors are not related.)

– ⟨Ax, y⟩ = ⟨x, A^{∗}y⟩ Proof:

⟨Ax, y⟩ = (Ax)^{∗}y = x^{∗}A^{∗}y = x^{∗}(A^{∗}y) = ⟨x, A^{∗}y⟩  (A.60)

If A = A^{∗}, it is a Hermitian matrix. A real Hermitian matrix is a symmetric matrix. The inverse of an invertible Hermitian matrix is also Hermitian, i.e., if A^{∗} = A, then (A^{−1})^{∗} = A^{−1}.

• Trace The trace of a square matrix A is the sum of its diagonal elements:

tr(A) = ∑_{i=1}^{n} a_{ii}  (A.61)

The properties of the trace:

– tr(cA) = c tr(A)

– tr(A^{T}) = tr(A)

– tr(A + B) = tr(B + A)

– tr(AB) = tr(BA)

– tr(A^{T} B) = tr(AB^{T})

– tr(R^{−1}AR) = tr(R^{−1}(AR)) = tr((AR)R^{−1}) = tr(A)

• Determinant

The determinant of a square matrix A is denoted by det(A) = |A|, and det(A) ≠ 0 if and only if it is full rank, i.e., rank(A) = n.

The properties of the determinant:

– det(I) = 1

– det(A^{T}) = det(A)

– det(A^{−1}) = det(A)^{−1}

– det(AB) = det(A) det(B)

– det(cA) = c^{n} det(A)

• Inverse

If A is an m×n full rank square matrix with m = n = r, then there exists an inverse matrix A^{−1} that satisfies AA^{−1} = A^{−1}A = I. The inverse A^{−1} does not exist if A is not square (m ≠ n) or full rank (r < m = n).

The properties of the inverse:

– (A^{−1})^{−1} = A

– (A^{T})^{−1} = (A^{−1})^{T}

– det(A^{−1}) = det(A)^{−1}

– (AB)^{−1} = B^{−1}A^{−1} For example, the solution of a linear equation system Ax = b of m

equations and n = m variables can be obtained as x = A^{−1}b.

A.4 Normal, Unitary, and Similar Matrices

• Commutative matrices Two square matrices A and B commute if AB = BA. Obviously all diagonal matrices commute.

• Normal matrix

A square matrix A is normal if it commutes with its conjugate transpose: A^{∗}A = AA^{∗}. If A^{∗} = A^{T} is real, then A^{T} A = AA^{T}.

Obviously unitary matrices (A^{∗} = A^{−1}), Hermitian matrices (A^{∗} = A), and skew-Hermitian matrices (A^{∗} = −A) are all normal. But there exist normal matrices that are not Hermitian.

• Unitary matrix

U = [u_{1}, ⋯, u_{n}] is a unitary matrix if its conjugate transpose is equal to its inverse U^{∗} = U^{−1}, i.e., U^{∗} U = I. When a unitary matrix \overline{U} = U is real, it becomes an orthogonal matrix, U^{T} = U^{−1}.

The column (or row) vectors of a unitary matrix A are orthonormal, i.e. they are both orthogonal and normalized:

⟨u_{i}, u_{j}⟩ = u_{i}^{T} \overline{u}_{j} = u_{j}^{∗}u_{i} = ∑_{k} u_{ik} \overline{u}_{jk} = δ_{ij} =^{△} \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (A.62)

Let λ and v be an eigenvalue and the corresponding eigenvector of U, i.e., Uv = λv, then we have

v^{∗}v = v^{∗}U^{∗}Uv = (Uv)^{∗}Uv = λ^{2}v^{∗}v  (A.63)

We see that λ^{2} = 1, i.e., |λ| = 1.

A Hermitian matrix A can be converted to a diagonal matrix Λ (or diagonalized) by a particular unitary matrix U:

U^{∗}AU = Λ = diag[λ_{1}, ⋯, λ_{n}]  (A.64)

where Λ is a diagonal matrix, i.e., all its off diagonal elements are 0.

• Similar matrices

Two matrices A and B are similar if they are related by A = P^{−1}BP. Similar matrices have the same determinant, trace, and eigenvalues.

A.5 Positive/Negative (Semi)-Definite Matrices

Given a Hermitian matrix A = A^{∗} and any non-zero vector x ≠ 0, we can construct a quadratic form x^{∗}Ax. The matrix A is said to be

• positive definite A > 0, if x^{∗}Ax > 0

• positive semi-definite A ≥ 0, if x^{∗}Ax ≥ 0

• negative definite A < 0, if x^{∗}Ax < 0

• negative semi-definite A ≤ 0, if x^{∗}Ax ≤ 0

• indefinite if there exists x and y such that x^{T} Ax < 0 < y^{T} Ay.

For example, consider the covariance matrix of a random vector x

Σ_{x} = E[(x − m_{x})(x − m_{x})^{∗}]  (A.65)

The corresponding quadratic form is

v^{∗}Σ_{x}v = v^{∗}E[(x − m_{x})(x − m_{x})^{∗}]v
= E[v^{∗}(x − m_{x})(x − m_{x})^{∗}v] = E(s^{2}) ≥ 0

where s = v^{∗}(x − m_{x}) is a scalar. Therefore Σ_{x} is positive semi-definite.

As A is Hermitian, its eigenvalues λ_{1}, ⋯, λ_{n} are real and its eigenvector matrix is unitary V^{−1} = V^{∗}, by which it can be diagonalized:

V^{−1}AV = V^{∗}AV = Λ, i.e. A = VΛV^{∗}  (A.66)

where vector y = V^{∗}x is a unitary transform of vector x.

For any x = [x_{1}, ⋯, x_{n}]^{T}, we have

x^{∗}Ax = x^{∗}(VΛV^{∗})x = (V^{∗}x)^{∗}Λ(V^{∗}x) = y^{∗}Λy = ∑_{i=1}^{n} λ_{i}|y_{i}|^{2}

= \{ \begin{matrix}> 0 is positive definte & if λ_{i} > 0 \\ ≥ 0 is positive semi-definte & if λ_{i} ≥ 0 \\ < 0 is negative definte & if λ_{i} < 0 \\ ≤ 0 is negative semi-definte & if λ_{i} ≤ 0 \end{matrix} (for all i)

Also, if some eigenvalues are positive and some others are negative, x^{∗}Ax may be either positive or negative depending on x , i.e., A is indefinite.

As the eigenvalues of A^{−1} are 1/λ_{i}, i = (1, ⋯, n), A^{−1} abd A share the same positive/negative definiteness.

A.6 Woodbury Matrix Identity and Sherman-Morrison Formula

The Woodbury matrix identity gives the inverse of an n × n square matrix A modified by a perturbation term UBV^{T}

(A + UBV^{T})^{−1} = A^{−1} − A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}  (A.67)

Proof:

(A + UBV^{T}) [A^{−1} − A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]

= [I − U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}] + [UBV^{T} A^{−1} − UBV^{T} A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]
= I + UBV^{T} A^{−1} − [U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1} + UBV^{T} A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}VA^{−1}]
= I + UBV^{T} A^{−1} − (U + UBV^{T} A^{−1}U)(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]
= I + UBV^{T} A^{−1} − UB(B^{−1} + V^{T} A^{−1}U)(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}
= I + UBV^{T} A^{−1} − UBV^{T} A^{−1} = I  (A.68)

Consider some special cases:

• If U = V = I, then we get

(A + B)^{−1} = A^{−1} − A^{−1}(A^{−1} + B^{−1})^{−1}A^{−1}  (A.69)

• If B = I, and let U = [u_{1}, ⋯, u_{m}] and V = [v_{1}, ⋯, v_{m}], then we get the inverse of a rank-n modified matrix:

(A+UV^{T})^{−1} = (A + ∑_{i=1}^{m} u_{i}v_{i}^{T} )^{−1} = A^{−1}−A^{−1}U(I+V^{T} A^{−1}U)^{−1}V^{T} A^{−1}

(A.70)

• More specially, if m = 1, U = u and V = v become vectors, then we get the Sherman-Morrison formula:

(A + uv^{T})^{−1} = A^{−1} − \frac{A^{−1}uv^{T} A^{−1}}{1 + v^{T} A^{−1}u}  (A.71)

A.7 Inverse and Determinant of Partitioned Symmetric Matrices

• Theorem 1 (inverse of a partitioned symmetric matrix)

Divide an n × n symmetric matrix A into four blocks

A = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ]  (A.72)

The inverse matrix B = A^{−1} can also be divided into four blocks:

B = A^{−1} = [ \begin{matrix}B_{11} & B_{12} \\ B_{21} & B_{22} \end{matrix} ]  (A.73)

Here we assume the dimensionalities of these blocks are:

– A_{11} and B_{11} are p × p,

– A_{22} and B_{22} are q × q,

– A_{12} = A_{21}^{T} and B_{12} = B_{21}^{T} are p × q with p + q = n. Then we have

B_{11} = (A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1} = A_{11}^{−1} + A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}A_{21}A_{11}^{−1} B_{22} = (A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1} = A_{22}^{−1} + A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}A_{12}A_{22}^{−1} B_{21} = −A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1} B_{12} = −A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}

(A.74)

Proof:

I_{n} = AA^{−1} = AB = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ] [ \begin{matrix}B_{11} & B_{12} \\ B_{21} & B_{22} \end{matrix} ]
= [ \begin{matrix}A_{11}B_{11} + A_{12}B_{21} & A_{11}B_{12} + A_{12}B_{22} \\ A_{21}B_{11} + A_{22}B_{21} & A_{21}B_{12} + A_{22}B_{22} \end{matrix} ] = [ \begin{matrix}I_{p} & 0 \\ 0 & I_{q} \end{matrix} ]

452  A Review of Linear Algebra

Equate each of the four blocks to get:

A_{11}B_{11} + A_{12}B_{21} = I_{p}, or B_{11} = A_{11}^{−1} − A_{11}^{−1}A_{12}B_{21}

A_{11}B_{12} + A_{12}B_{22} = 0, or B_{12} = −A_{11}^{−1}A_{12}B_{22}
A_{21}B_{11} + A_{22}B_{21} = 0, or B_{21} = −A_{22}^{−1}A_{21}B_{11}
A_{21}B_{12} + A_{22}B_{22} = I_{q} or B_{22} = A_{22}^{−1} − A_{22}^{−1}A_{21}B_{12}

(A.75)

Plug B_{21} into B_{11} to get:

B_{11} = A_{11}^{−1} + A_{11}^{−1}A_{12}A_{22}^{−1}A_{21}B_{11}  (A.76)

Solve for B_{11} to get:

B_{11} = (A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}  (A.77)

Applying theorem 1 to this expression, we also get the other expression in the theorem. Similarly we can get

B_{22} = (A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}
B_{21} = −A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}
B_{12} = −A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12}^{−1})^{−1}

• Theorem 2 (Determinant of a partitioned symmetric matrix)

|A| = |[ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ]| = |A_{22}| |A_{11}−A_{12}A_{22}^{−1}A_{21}| = |A_{11}| |A_{22}−A_{21}A_{11}^{−1}A_{12}|

(A.78)

Proof:

A = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ] = [ \begin{matrix}A_{11} & 0 \\ A_{21} & I \end{matrix} ] [ \begin{matrix}I & A_{11}^{−1}A_{12} \\ 0 & A_{22} − A_{21}A_{11}^{−1}A_{12} \end{matrix} ]
= [ \begin{matrix}I & A_{12} \\ 0 & A_{22} \end{matrix} ] [ \begin{matrix}A_{11} − A_{12}A_{22}^{−1}A_{21} & 0 \\ A_{22}^{−1}A_{21} & I \end{matrix} ]

The theorem is proved as we also know that

|BC| = |B| |C|  (A.79)

and

| \begin{matrix}B & 0 \\ X & C \end{matrix} | = | \begin{matrix}B & X \\ 0 & C \end{matrix} | = |B| |C|  (A.80)

A.8 Unitary Transform

Given any unitary matrix A satisfying AA^{∗} = A^{∗}A = I, we can define a unitary transform of a vector x = [x_{1}, ⋯, x_{n}]^{T}:

y = [ y_{1} ; ⋮ ; y_{n} ] = A^{∗}x = [  \overline{a}_{1}^{T} ; ⋮ ; \overline{a}_{n}^{T}  ] [ x_{1} ; ⋮ ; x_{n} ] , (forward transform)

x = [ x_{1} ; ⋮ ; x_{n} ] = Ay = [ a_{1} ⋯ a_{n} ] [ y_{1} ; ⋮ ; y_{n} ] = ∑_{i=1}^{n} y_{i} a_{i} (inverse transform)

(A.81)

When A = \overline{A} is real, A^{−1} = A^{T} is an orthogonal matrix and the corresponding transform is an orthogonal transform.

The first equation above is the forward transform and can be written in component form as:

y_{i} = \overline{a}_{i}^{T} x = ⟨x, a_{i}⟩ = ∑_{j=1}^{n} x_{j}\overline{a}_{ij}, (i = 1, ⋯, n)  (A.82)

The transform coefficient is an inner product y_{i} = ⟨x, a_{i}⟩, representing the projection of vector x onto the ith column vector a_{i} of the transform matrix A. The second equation is the inverse transform and can also be written in component form as:

x_{j} = ∑_{i=1}^{n} a_{ji} y_{i}, (j = 1, ⋯, n)  (A.83)

By this transform, vector x is represented as a linear combination (weighted sum) of the n column vectors a_{i}, a_{2}, ⋯, a_{n} of matrix A. Geometrically, x is a point in the n-dimensional space spanned by these n orthonormal basis vectors. Each coefficient (coordinate) y_{i} is the projection of x onto the corresponding basis vector a_{i}.

As the n-dimensional space can be spanned by the column vectors of any n×n unitary (orthogonal) matrix, a vector x in the space can be represented by any of such matrices, each defining a different transform.

Examples:

• When A = I = [⋯, e_{i}, ⋯] is an identity matrix, we have

x = ∑_{i=1}^{n} y_{i}a_{i} = ∑_{i=1}^{n} x_{i}e_{i}  (A.84)

where e_{i} = [0, ⋯, 0, 1, 0, ⋯, 0]^{T} is the ith column of I with the ith element equal 1 and all other 0.

• When a_{m,n} = w[m, n] = e^{−j2πmn/N}, the corresponding transform is discrete Fourier transform. The nth column vector w_{n} of the transform matrix W = [w_{0}, ⋯, w_{N−1}] represents a sinusoid of a frequency nf_{0}, and the corresponding complex coordinate y_{n} = (x, w_{n}) represents the magnitude |y_{n}| and phase ∠y_{n} of this nth frequency component. The Fourier transform y = Wx represents a rotation of the coordinate system.

A unitary (orthogonal if real) transform y = A x can be interpreted geometrically as the rotation of the vector x about the origin, or equivalently, the representation of the same vector in a rotated coordinate system. A unitary transform y = A x does not change the inner product. Let u = A^{∗}x and v = A^{∗}y be the unitary transforms of vectors x and t, then the inner product of these two vectors is preserved:

⟨u, v⟩ = ⟨A^{∗}u, A^{∗}v⟩ = x^{∗}AA^{∗}y = x^{∗}y = ⟨x, y⟩  (A.85)

In particular, if x = y, we see that the vector norm is preserved by the unitary transform:

||u||^{2} = ⟨u, u⟩ = ⟨x, x⟩ = ||x||^{2}  (A.86)

This is the Parseval’s identity that indicates that the norm (length) of a vector is preserved under any unitary transform. If X is interpreted as a signal, then its length ||x||^{2} = ||y||^{2} represents the total energy or information contained in the signal, which is conserved during any unitary transform. However, some other features of the signal may change, e.g., the signal may be decorrelated and its total energy redistributed among its components after the transform, which may be desirable in many applications.

If x is a random vector with mean vector m_{x} and covariance matrix Σ_{x}:

m_{x} = E(x), Σ_{x} = E(xx^{∗}) − m_{x}m_{x}^{∗}  (A.87)

then its transform y = A^{∗}x has the following mean vector and covariance matrix:

m_{y} = E(y) = E(A^{∗}x) = A^{∗}E(x) = A^{∗}m_{x}  (A.88)
Σ_{y} = E(yy^{∗}) − m_{y}m_{y}^{∗} = E[(A^{∗}x)(A^{∗}x)^{∗}] − (A^{∗}m_{x})(A^{∗}m_{x})^{∗}
= E[A^{∗}(xx^{∗})A] − A^{∗}m_{x}m_{x}^{∗}A = A^{∗}[E(xx^{∗}) − m_{x}m_{x}^{∗}]A
= A^{∗}Σ_{x}A

In general the unitary transform of any square matrix A by a unitary matrix R is

B = R^{∗}AR  (A.89)

A.9 Eigenvalue Decomposition

For any n × n square matrix A, if there exist a vector v and a scalar λ such that the following eigenequation holds:

Av = λv,  (A.90)

then λ and v are called the eigenvalue and eigenvector of matrix A, respectively. In other words, the linear transformation of vector v by A has the same effect of scaling the vector by factor λ. (Note that for an m × n non-square matrix A with m ≠ n, Av is an m-D vector but λv is n-D vector, i.e., no eigenvalues and eigenvectors are defined.)

Given Av = λv, we also have Acv = λcv for any scalar constant c, i.e., the eigenvector v is not unique but up to any scaling factor. For the uniqueness of v, we typically keep it normalized so that ||v|| = 1.

To obtain λ, we rewrite the above equation as

(λI − A)v = 0  (A.91)

For this homogeneous equation system to have non-zero solutions for v, the determinant of its coefficient matrix has to be zero:

det(λI − A) = |λI − A| = 0  (A.92)

This is the characteristic polynomial equation of matrix A. Solving this nth order equation of λ, we get n eigenvalues {λ_{1}, ⋯, λ_{n}}. Substituting each λ_{i} back into the homogeneous equation system, we get the corresponding eigenvector v_{i}. We can put all n eigen-equations Av_{i} = λ_{i}v_{i} together and obtain the more compact form:

AV = A[v_{1}, ⋯, v_{n}] = [λ_{1}v_{1}, ⋯, λ_{n}v_{n}] = [v_{1}, ⋯, v_{n}] [ \begin{matrix}λ_{1} & 0 & ⋯ & 0 \\ 0 & λ_{2} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & λ_{n} \end{matrix} ] = VΛ

(A.93)

where we have defined

Λ = diag[λ_{1}, ⋯, λ_{n}] and V = [v_{1}, ⋯, v_{n}]  (A.94)

The eigen-equations can be written in some alternative forms:

A = VΛV^{−1}, or V^{−1}AV = Λ  (A.95)

In the first form, A = VΛV^{−1} is expressed as a product of three matrices, called eigenvalue decomposition of the matrix; in the second form, A is diagonalized by its eigenvector matrix V to become a diagonal matrix, its eigenvalue matrix Λ.

Given Av = λv, we have the following:

• A^{T} has the same eigenvalues and eigenvectors as A.

Proof: As a matrix A and its transpose A^{T} have the same determinant, they have the same characteristic polynomial:

|A − λI| = |(A − λI)^{T} | = |A^{T} − λI|  (A.96)

therefore they have the same eigenvalues and eigenvectors.

• The eigenvalues and eigenvectors of A^{∗} are the complex conjugate of the eigenvalues and eigenvectors as A.

• A^{T} A has the same eigenvectors as A, but its eigenvalues are λ^{2}.

Proof:

A^{T} Av = A^{T} λv = λ^{2}v  (A.97)

• A^{k} has the same eigenvectors as A, but its eigenvalues are {λ_{1}^{k}, ⋯, λ_{n}^{k}},

where k is a positive integer.

Proof:

A^{2}v = AAv = Aλv = λ^{2}v  (A.98)

This result can be generalized to

A^{k}v = λ^{k}v  (A.99)

• In particular when k = −1, i.e., the eigenvalues of A^{−1} are {λ_{1}^{−1}, ⋯, λ_{n}^{−1}}. Proof: Left multiplying A^{−1} on both sides of Av_{i} = λv_{i} we get

A^{−1}Av_{i} = Iv_{i} = v_{i} = λ_{i}A^{−1}v_{i}  (A.100)

Dividing both sides by λ we get

A^{−1}v_{i} = \frac{1}{λ}_{i}v_{i} = λ_{i}^{−1}v_{i}  (A.101)

• The eigenvalues of a matrix are invariant under any unitary transform B =

R^{∗}AR, where R is unitary, satisfying R^{−1} = R^{∗}, or R^{∗}R = RR^{∗} = I Proof: Let Λ = diag(λ_{1}, ⋯, λ_{n}) and V = [v_{1}, ⋯, v_{n}] be the eigenvalue and

eigenvector matrices of a square matrix A:

AV = VΛ  (A.102)

and Σ = diag(σ_{1}, ⋯, σ_{n}) and U = [u_{1}, ⋯, u_{n}] be the eigenvalue and eigenvector matrices of B = R^{∗}AR, a unitary transform of A:

BU = (R^{∗}AR) U = UΣ  (A.103)

Left-multiplying R on both sides we get the eigenequation of A

RR^{∗}ARU = A(RU) = (RU)Σ  (A.104)

We see that A and B = R^{∗}AR have the same eigenvalues Σ = Λ and their eigenvector matrices are related by V = RU or U = R^{∗}V.

• Given all eigenvalues λ_{1}, ⋯, λ_{n} of a matrix A, its trace and determinant can be obtained as

tr(A) = ∑_{k=1}^{n} λ_{k},  det(A) = ∏_{k=1}^{n} λ_{k}  (A.105)

• The spectrum of an n × n square matrix A is the set of its eigenvalues {λ_{1}, ⋯, λ_{n}}. The spectral radius of A, denoted by ρ(A), is the maximum of the absolute values of the elements in the spectrum:

ρ(A) = max(|λ_{1}|, ⋯, |λ_{n}|)  (A.106)

where |z| = \sqrt{x^{2} + y^{2}} is the modulus of a complex number z = x + jy. If all eigenvalues are sorted such that |λ_{1}| ≥ ⋯ ≥ |λ_{n}| then ρ(A) = |λ_{1}| = |λ_{max}|. As the eigenvalues of A^{−1} are {1/λ_{max}, ⋯, 1/λ_{min}}, ρ(A^{−1}) = 1/|λ_{min}.

• If A and B are similar, i.e.,

B = P^{−1}AP  (A.107)

then they have the same eigenvalues.

Proof: Let Λ and V be the eigenvalue and eigenvector matrices of A:

AV = VΛ,  A = VΛV^{−1}  (A.108)

then we have

B = P^{−1}AP = P^{−1}VΛV^{−1}P = UΛU^{−1}  (A.109)

i.e., Λ and U = P^{−1}V are the eigenvalue and eigenvector matrices of B = P^{−1}AP.

• If A is Hermitian A^{∗} = Ā^{T} = A (symmetric A^{T} = A if real) (e.g., the covariance matrix of a random vector), then all of its eigenvalues λ̄_{i} = λ_{i} are real, and all of its eigenvectors are orthogonal, v_{i}^{∗}v_{j} = 0 (i ≠ j) i.e., V^{−1} = V^{∗}. And we further have

A = VΛV^{−1} = VΛV^{T} = VΛ^{1/2}Λ^{1/2}V^{T} = UU^{T}  (A.110)

where U = VΛ^{1/2}.

Proof:

Let λ and v be an eigenvalue and the corresponding eigenvector of a Hermitian matrix A = A^{∗}, i.e., Av = λv, then we have

(Av)^{∗}v = (λv)^{∗}v = λ̄v^{∗}v = λ̄ ||v||^{2}
= v^{∗}A^{∗}v = v^{∗}Av = v^{∗}λv = λ ||v||^{2}  (A.111)

i.e., λ̄ = λ is real. We also have

v_{i}^{∗}Av_{j} = v_{i}^{∗}λ_{j}v_{j} = λ_{j}v_{i}^{∗}v_{j}
= (v_{j}^{∗}Av_{i})^{∗} = (v_{j}^{∗}λ_{i}v_{i})^{∗} = λ̄_{i}v_{i}^{∗}v_{j} = λ_{i}v_{i}^{∗}v_{j}  (A.112)

i.e.,

λ_{j}v_{i}^{∗}v_{j} = λ_{i}v_{i}^{∗}v_{j}, or (λ_{i} − λ_{j}) v_{i}^{∗}v_{j} = 0  (A.113)

As λ_{i} ≠ λ_{j}, we get v_{i}^{∗}v_{j} = 0, i.e., the eigenvectors corresponding to different eigenvalues are orthogonal.

QED

When all eigenvectors are normalized v_{i}^{∗}v_{i} = ||v_{i}||^{2} = 1, they become orthonormal

⟨v_{i}, v_{j}⟩ = v_{i}^{∗}v_{j} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (A.114)

i.e., the eigenvector matrix V = [v_{1}, ⋯, v_{n}]^{T} is unitary (orthogonal if A is real):

V^{−1} = V^{∗} i.e. V^{∗}V = I  (A.115)

and we have

V^{−1}AV = V^{∗}AV = Λ,  (A.116)

Left and right multiplying by V and V^{∗} = V^{−1} respectively on the two sides, we get

A = VΛV^{∗} = [v_{1}, ⋯, v_{n}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{n} \end{matrix} ] [ v_{1}^{∗} ; ⋮ ; v_{n}^{∗} ] = ∑_{i=1}^{n} λ_{i}v_{i}v_{i}^{∗}

(A.117)

This is the spectral theorem indicating that A can be written as a linear combination of n matrices v_{i}v_{i}^{∗} weighted by λ_{i} (n = 1, ⋯, n).

The significance of this property is that a linear operation y = Ax applied to vector x can be mapped to a new vector space in which the operations of the components are independent of each other. Consider

y = Ax = VΛV^{∗}x  (A.118)

Pre-multiplying V^{∗} on both sides we get

y′ = V ∗ y = ΛV^{∗}x = Λx′  (A.119)

where we have defined x′ = V^{∗}x and y′ = V^{∗}y, the unitary transform of x and y, respectively. We see that for the ith component we have the following, independent of all other components:

y′_{i} = λ_{i}x′_{i}  (A.120)

• The entries on the diagonal of an upper (or lower) triangular matrix are its

eigenvalues.

Proof: Let A be an upper triangular matrix with a_{ij} = 0 for all i > j. The eigenvalues of A are the roots of the following homogeneous characteristic equations:

det(A − λI) = ∏_{i=1}^{n}(a_{ii} − λ) = 0  (A.121)

The first equal sign is due to the fact that A−λI is also an upper-triangular matrix, and the determinant of an upper-triangular matrix is the product of all its diagonal entries. We therefore see that each diagonal entry a_{ii}, as a root of the characteristic equation, is also an eigenvalue of A.

QED

• Similar matrices have the same eigenvalues.

Proof: Let λ and v be an eigenvalue and the corresponding eigenvector of A satisfying Av = λv, and B = P^{−1}AP be a similar matrix of A. We have A = PBP^{−1} and

Av = PBP^{−1}v = λv, i.e., BP^{−1}v = λP^{−1}v  (A.122)

In other words, λ is also an eigenvalue of B with the corresponding eigenvector P^{−1}v.

QED

• All eigenvalues of a stochastic matrix P are no greater than 1. A stochastic matrix is a matrix of which component P_{ij} is the probability for a state transition of a Markov process (chain) from s_{i} to s_{j}, i.e., 0 ≤ P_{ij} ≤ 1 and ∑_{j} P_{ij} = 1.

Proof: First, as P1 = 1, λ = 1 is one of the eigenvalues of P. Next let λ and v be an eigenvalue and the corresponding eigenvector of P, i.e., Pv = λv, and let |v_{k}| ≥ max{|v_{1}|, ⋯, |v_{n}|}. The kth row of the eigenequation eqwuation is

|λv_{k}| ≤ |λ| |v_{k}| = | ∑_{j=1}^{n} P_{kj}v_{j}| ≤ ∑_{j=1}^{n} P_{kj} |v_{j}| ≤ |v_{k}| ∑_{j=1}^{n} P_{kj} = |v_{k}| (A.123)

i.e., |λ| ≤ 1.

QED

A.10 Generalized Eigenvalue Problem

The generalized eigenvalue problem of two symmetric matrices A = A^{T} and B = B^{T} is to find a scalar λ_{i} and the corresponding vector v_{i} for the following generalized eigen equation to hold:

Av_{i} = λ_{i}Bv_{i},  (i = 1, ⋯, n)  (A.124)

or in matrix form

AV = BVΛ  (A.125)

The eigenvalue and eigenvector matrices Λ and V can be found in the following steps.

• Solve the eigenvalue problem of B to find its diagonal eigenvalue matrix Λ_{B} and orthogonal eigenvector matrix V_{B} = (V_{B}^{T})^{−1} so that

BV_{B} = V_{B}Λ_{B},  or V_{B}^{−1}BV_{B} = V_{B}^{T}BV_{B} = Λ_{B}  (A.126)

• Left and right multiplying both sides of the second equation above by Λ^{−1/2} (whitening) we get

Λ_{B}^{−1/2}(V_{B}^{T}BV_{B})Λ_{B}^{−1/2} = Λ_{B}^{−1/2}Λ_{B}Λ_{B}^{−1/2} = I  (A.127)

We define

V′_{B} = V_{B}Λ_{B}^{−1/2}  (A.128)

and the above becomes

(V′_{B})^{T} BV′_{B} = I  (A.129)

Note that V′_{B} is not orthogonal

(V′)_{B}^{−1} = (V_{B}Λ_{B}^{−1/2})^{−1} = Λ_{B}^{1/2}V_{B}^{−1} = Λ_{B}^{1/2}V_{B}^{T} ≠ Λ_{B}^{−1/2}V_{B}^{T} = V_{B}^{T}

(A.130)

• Apply the same transform to A:

(V′_{B})^{T} AV′_{B} = (Λ_{B}^{−1/2}V_{B}^{T})A(V_{B}Λ_{B}^{−1/2}) = A′  (A.131)

Note that A′ is symmetric as well as A:

A^{′T} = (V_{B}^{′T} AV′_{B})^{T} = V_{B}^{′T} AV′_{B} = A′  (A.132)

• Diagonalize A′

As A′ is symmetric,, it can be diagonalized by its orthogonal eigenvector matrix V_{A}:

V_{A}^{T}A′V_{A} = Λ  (A.133)

i.e.,

V_{A}^{T}(Λ_{B}^{−1/2}V_{B}^{T}AV_{B}Λ_{B}^{−1/2})V_{A} = (V_{A}^{T}Λ_{B}^{−1/2}V_{B}^{T})A(V_{B}Λ_{B}^{−1/2}V_{A})
= V^{T} AV = Λ

where we have defined

V = V_{B}Λ_{B}^{−1/2}V_{A}  (A.134)

which is not orthogonal:

V^{−1} = (V_{B}Λ_{B}^{−1/2}V_{A})^{−1} = V_{A}^{−1}Λ_{B}^{1/2}V_{B}^{−1} = V_{A}^{T}Λ_{B}^{1/2}V_{B}^{T} ≠ V_{A}^{T}Λ_{B}^{−1/2}V_{B}^{T} = V^{T}

(A.135)

• This V also diagonalizes B:

V^{T} BV = (V_{B}Λ_{B}^{−1/2}V_{A})^{T} B(V_{B}Λ_{B}^{−1/2}V_{A}) = V_{A}^{T}Λ_{B}^{−1/2}(V_{B}^{T}BV_{B})Λ_{B}^{−1/2}V_{A}
= V_{A}^{T}Λ_{B}^{−1/2}Λ_{B}Λ_{B}^{−1/2}V_{A} = V_{A}^{T}V_{A} = I

• Now we have

\{ \begin{matrix}V^{T} AV = Λ \\ V^{T} BV = I \end{matrix}  (A.136)

Right multiplying both sides of the second equation by Λ we get V^{T} BVΛ = Λ, then equating the left-hand side to that of the first equation, we get

V^{T} AV = V^{T} BVΛ, i.e. AV = BVΛ  (A.137)

i.e., Λ and V are the eigenvalue and eigenvector matrices of the generalized eigenvalue problem. Note, however, as shown above, V is not orthogonal.

A.11 Rayleigh Quotient

The Rayleigh quotient R(A) of a hermitian matrix A = A^{∗} (symmetric if A is real) is defined as

R(A) = \frac{w^{∗}Aw}{w^{∗}w}  (A.138)

where w is any non-zero vector. In particular, if w = v_{i} is the ith eigenvector of A, then the Rayleigh quotient is the corresponding to eigenvalue λ_{i}:

R(A) = \frac{v_{i}^{∗}Av_{i}}{v_{i}^{∗}v_{i}} = \frac{λ_{i}v_{i}^{∗}v_{i}}{v_{i}^{∗}v_{i}} = λ_{i}  (A.139)

The Rayleigh quotient of two symmetric matrices A and B is a function of a vector w defined as:

R(w) = \frac{w^{T} Aw}{w^{T} Bw}  (A.140)

To find the optimal w corresponding to the extremum (maximum or minimum) of R(w), we find its derivative with respect to w:

\frac{d}{dw} R(w) = \frac{2Aw(w^{T} Bw) − 2Bw(w^{T} Aw)}{(w^{T} Bw)^{2}}  (A.141)

Setting it to zero we get

Aw(w^{T} Bw) = Bw(w^{T} Aw), i.e. Aw = \frac{w^{T} Aw}{w^{T} Bw} Bw = R(w)Bw = λBw

(A.142)

The second equation can be recognized as a generalized eigenvalue problem with λ = R(w) being the eigenvalue and w the corresponding eigenvector. Solving this we get the vector w corresponding to the maximum/minimum eigenvalue λ = R(w), which maximizes/minimizes the Rayleigh quotient.

A.12 Normal Matrices and Diagonalizability

Theorem: The product of two unitary matrices is unitary.

Proof: Let U and V be unitary, i.e., U^{∗} = U^{−1} and V^{∗} = V^{−1}, then UV is unitary:

(UV)^{∗} = V^{∗}U^{∗} = V^{−1}U^{−1} = (UV)^{−1}  (A.143)

Theorem: Two square matrices A and B are simultaneously diagonalizable if and only if they commute.

Proof

• Let A and B be simultaneously diagonalizable by R

R^{−1}AR = Λ_{A},  R^{−1}BR = Λ_{B}  (A.144)

then

AB = (RΛ_{A}R^{−1})(RΛ_{B}R^{−1}) = (RΛ_{A}Λ_{B}R^{−1}) = (RΛ_{B}Λ_{A}R^{−1}) = (RΛ_{B}R^{−1})(RΛ_{A}R^{−1}) = BA

(A.145)

• Let A and B commute, i.e., AB = BA. Assuming u is an eigenvector of A corresponding to eigenvalue λ, i.e., Au = λu, then

ABu = B(Au) = λBu  (A.146)

we see that Bu is also an eigenvector of A corresponding to the same eigenvalue λ, i.e., Bu must be a scaled version of u (in the same 1-D space): Bu = γu, i.e., u is also an eigenvector of B.

Theorem: A matrix is normal if and only if it is unitarily diagonalizable. Proof

• If A is unitarily diagonalizable:

AU = UΛ,  U^{−1}AU = U^{∗}AU = Λ,  A = UΛU^{∗}  (A.147)

where U^{∗} = U^{−1} is unitary and Λ is a diagonal matrix satisfying Λ^{∗}Λ = ΛΛ^{∗}, then A is normal:

AA^{∗} = (UΛU^{∗})(UΛU^{∗})^{∗} = (UΛU^{∗})(UΛ^{∗}U^{∗}) = UΛΛ^{∗}U^{∗}
= UΛ^{∗}ΛU^{∗} = (UΛ^{∗}U^{∗})(UΛU^{∗}) = A^{∗}A

• If A is normal, then it is diagonalizable by a unitary matrix. First we show any matrix A can be written as

A = B + iC  (A.148)

where

B = \frac{1}{2}(A + A^{∗}) = B^{∗}, C = −\frac{1}{2}(A − A^{∗}) = C^{∗},  (A.149)

are both Hermitian, and diagonalizable by a unitary matrix. As A is normal, we have

0 = AA^{∗} − A^{∗}A = (B + iC)(B − iC) − (B − iC)(B + iC) = 2i(CB − BC)

(A.150)

We see that CB = BC, i.e., B and C commute, and they can be simultaneously diagonalized by some unitary matrix U:

U^{∗}BU = Λ_{B},  U^{∗}CU = Λ_{C},  (A.151)

and so can A = B + iC:

U^{∗}AU = U^{∗}(B + iC)U = Λ_{B} + iΛ_{C} = Λ_{A}  (A.152)

Q.E.D.

If A is normal, i.e., it commutes with its conjugate transpose A^{∗}A = AA^{∗}, then A and A^{∗} can simultaneously diagonalized by their unitary eigenvector matrix U:

A = UΛU^{∗}, A^{∗} = U\overline{Λ}U^{∗}  (A.153)

A.13 Centering Matrix

The centering matrix is defined as

C_{n} = I_{n} − \frac{1}{n}11^{T}  (A.154)

where 1 = [1, ⋯, 1]^{T} and 11^{T} is an n × n matrix of all 1’s. For example,

C_{1} = 0;  C_{2} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] − \frac{1}{2} [ \begin{matrix}1 & 1 \\ 1 & 1 \end{matrix} ] = \frac{1}{2} [ \begin{matrix}1 & −1 \\ −1 & 1 \end{matrix} ];  (A.155)

C_{3} = [ \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ] − \frac{1}{3} [ \begin{matrix}1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{matrix} ] = \frac{1}{3} [ \begin{matrix}2 & −1 & −1 \\ −1 & 2 & −1 \\ −1 & −1 & 2 \end{matrix} ]  (A.156)

When C_{n} is applied to an n-D vector v = [v_{1}, ⋯, v_{n}]^{T}, we get

C_{n}v = v − \frac{1}{n} (11^{T} ) v = v − \frac{1}{n} ∑_{i=1}^{n} v_{i}1 = v − v̄1  (A.157)

where v̄ is the mean of all components of v:

v̄ = \frac{1}{n}1^{T} v = \frac{1}{n} ∑_{i=1}^{n} v_{i}  (A.158)

We see that the mean of v is removed from the resulting vector C_{n}v. Example: v = [3, −2, 5]^{T}, with mean v̄ = (3 − 2 + 5)/3 = 2,

u = C_{3}v = \frac{1}{3} [ \begin{matrix}2 & −1 & −1 \\ −1 & 2 & −1 \\ −1 & −1 & 2 \end{matrix} ] [ \begin{matrix}3 \\ −2 \\ 5 \end{matrix} ] = [ \begin{matrix}3 \\ −2 \\ 5 \end{matrix} ] − [ \begin{matrix}2 \\ 2 \\ 2 \end{matrix} ] = [ \begin{matrix}1 \\ −4 \\ 3 \end{matrix} ]

(A.159)

The mean of the resulting vector u is ū = 0. In particular, if v = 1, then

C_{n}v = C_{n}1 = [I − \frac{1}{n}11^{T} ] 1 = 1 − \frac{1}{n}11^{T} 1 = 1 − \frac{1}{n}1n = 0  (A.160)

i.e., C_{n}1 = 0 1. We therefore see that λ = 0 and v = 1 are the eigenvalue and corresponding eigenvector of C_{n}. Also, consider any zero-mean vector v = [v_{1}, ⋯, v_{n}] satisfying ∑_{i=1}^{n} v_{i} = 1^{T} v = 0 with n−1 degrees of freedom, we have

C_{n}v = [I − \frac{1}{n}11^{T} ] v = v − \frac{1}{n}11^{T} v = v  (A.161)

i.e., C_{n}v = 1 v. We therefore see that λ = 1 and any v satisfying 1^{T} v = 0 are the eigenvalue of multiplicity n − 1 and the corresponding eigenvector of C_{n}.

The certering matrix C_{n} has the following properties:

• Idempotence: C_{n}^{k} = C_{n}

C_{n}^{2} = C_{n}C_{n} = (I_{n} − \frac{1}{n}11^{T} ) (I_{n} − \frac{1}{n}11^{T} ) = I_{n} − \frac{2}{n}11^{T} + \frac{n}{n^{2}} 11^{T} = C_{n}

(A.162)

Once the mean of vector v is removed, its mean is zero and subsequent removal of mean has not effect.

• C_{n} has eigenvalue λ = 1 of multiplicity n − 1 and eigenvalue λ = 0 of multiplicity 1.

• C_{n} is singular, its inverse does not exist. Once the mean of vector v is removed, it cannot be reconstructed by an inverse process.

A.14 Singular Value Decomposition

The eigenvalue decomposition of a square matrix A = VΛV^{−1} is of great importance and widely used. However, for an m × n non-square matrix A, no eigenvalues and eigenvector exist. In this case, we can still find its singular values and the corresponding left and right singular vectors, and then carry out singular value decomposition (SVD).

Theorem: An m × n matrix A of rank r ≤ min(m, n) can be diagonalized by two orthogonal matrices U_{m×m} and V_{n×n}:

U^{T} AV = Σ  or A = UΣV^{T}  (A.163)

where

• U = [u_{1}, …, u_{m}] is an m×m matrix composed of the orthogonal eigenvectors of the symmetric matrix AA^{T}, also called the left singular vectors of A:

U^{T} (AA^{T})U = Λ  or AA^{T} = UΛU^{T}  (A.164)

where Λ = diag(λ_{1}, ⋯, λ_{n}) is the e igenvalue matrix of AA^{T}.

• V = [v_{1}, …, v_{n}] is an n × n matrix composed of the orthogonal eigenvectors of the symmetric matrix A^{T} A, also called the right singular vectors of A:

V^{T} (A^{T} A)V = Λ  or A^{T} A = VΛV^{T}  (A.165)

Note that A^{T} A and AA^{T} have the same eigenvalue matrix Λ (see proof below).

• Σ = Λ^{1/2} is an m × n diagonal matrix of r non-zero values σ_{i} = \sqrt{λ_{i}} (i = 1, …, r), the square roots of the eigenvalues of AA^{T} or A^{T} A, defined as the singular values of A:

Λ^{1/2} = [ \begin{matrix}√ & \overline{λ_{1}} \\ & . & . & . \\ √ & \overline{λ_{r}} \end{matrix}  ] = [ \begin{matrix}σ_{1} \\ & . & . & . \\ & & & & σ_{r} \end{matrix}  ] = Σ (A.166)

Proof:

Pre-multiplying A on both sides of the eigenequation of A^{T} A:

(A^{T} A)V = VΛ  (A.167)

we get

(AA^{T})AV = AVΛ,  (A.168)

Note that A^{T} A and AA^{T} have the same eigenvalue matrix Λ, and AV is the eigenvector matrix of AA^{T}. As the columns (or rows) of AV are not normalized, AV is not orthogonal:

(AV)^{T} (AV) = V^{T} A^{T} AV = V^{T} VΛ = Λ ≠ I  (A.169)

If we post-multiply AV by Λ^{−1/2}, its ith column is scaled by 1/\sqrt{λ_{i}} (i = 1, ⋯, n) and thereby normalized, the resulting matrix becomes orthogonal:

(AVΛ^{−1/2})^{T} (AVΛ^{−1/2}) = Λ^{−1/2}V^{T} A^{T} AVΛ^{−1/2}
= Λ^{−1/2}V^{T} VΛΛ^{−1/2} = Λ^{−1/2}ΛΛ^{−1/2} = I

We denote this matrix composed of normalized orthogonal eigenvectors of AA^{T} by

AVΛ^{−1/2} = U  (A.170)

Post-multiplying (VΛ^{−1/2})^{−1} = Λ^{1/2}V^{−1} on both sides, we get the SVD of A:

A = UΛ^{1/2}V^{−1} = UΛ^{1/2}V^{T} = UΣV^{T}  (A.171)

Q.E.D.

The SVD is illustrated graphically below for the two cases of m < n and m > n:

The component form of AVΛ^{−1/2} = U above is

Av_{i} \frac{1}{\sqrt{λ_{i}}} = u_{i} (i = 1, ⋯, n)  (A.172)

relating the eigenvectors v_{i} of A^{T} A to the eigenvectors u_{i} of AA^{T}. We further consider some special cases:

• If A^{T} = A^{−1} is unitary, i.e., A^{T} A = AA^{T} = I, its eigenvalue matrix is Λ = I, and all singular values are unity σ_{i} = 1 for all i = 1, ⋯, n.

• If A^{T} = A is symmetric, i.e., AA^{T} = A^{T} A = A^{2}, and its singular value decompositon is A = UΣV^{T}, i.e.,

AA^{T} = UΣ^{2}U^{T} , A^{T} A = VΣ^{2}V  (A.173)

On the other hand, the eigenvalue decomposition of A^{2} = A^{T} A = AA^{T} can be written as:

A^{2} = WΛ^{2}W  (A.174)

where Λ is the eigenvalue matrix of A. We see that Λ^{2} = Σ^{2}, i.e., the singular values are the absolute values of its eigenvalues σ_{i} = \sqrt{λ_{i}^{2}} = |λ_{i}| (i = 1, ⋯, n), and the left and right eigenvectors are the same as the eigenvectors U = V = W.

• If A is normal satisfying A^{T} A = AA^{T} (e.g., symmetric, Hermitian, unitary, etc.), both A and A^{T} can be simultaneously diagonalized by their unitary eigenvector matrix U = V, i.e.,

Au_{i} = λ_{i}u_{i}, Au_{i} = \overline{λ_{i}}u_{i}, (i = 1, ⋯, n)  (A.175)

or in matrix form

A = UΛU^{T} , A^{T} = U\overline{Λ}U^{T}  (A.176)

we have

AA^{T} = (UΛU^{T})(U\overline{Λ}U^{T}) = UΛ\overline{Λ}U^{T} = U|Λ|^{2}U^{T}  (A.177)

i.e., the singular values of any norm matrix A are simply the modulus of its eigenvalues σ_{i}(A) = |λ_{i}(A)| (i = 1, ⋯, n).

The m × n matrix A = UΣV^{T} can be considered as a linear transformation that converts a vector x ∈ ℂ^{n} to another vector y ∈ ℂ^{m} in three steps:

y = Ax = UΣV^{T} x  (A.178)

1. Rotate vector x by the orthogonal matrix V^{T}:

y_{1} = V^{T} x.  (A.179)

2. Scale each dimension Y_{k} of y_{1} by a factor of σ_{k} (k = 1, …, r):

y_{2} = Σ y_{1} = ΣV^{T} x.  (A.180)

3. Rotate vector y_{2} by the orthogonal matrix U:

y = Uy_{2} = UΣV^{T} x = Ax.  (A.181)

The figure below illustrates the transformation of the three vertices of a triangle in 2-D space by a matrix A = UΣV^{∗}, which first rotates the vertices by 45 degrees CCW, scale horizontally and vertically by a factor of 3 and 2, respectively, and then rotate CW by 30 degrees.

The SVD has many applications, two of which are considered below.

The SVD equation

U^{T} AV = Λ^{1/2} = Σ  (A.182)

can be considered as the forward SVD transform. Pre-multiplying U and postmultiplying V^{T} on both sides, we get the inverse transform:

A = UΣV^{T} = [u_{1}, · · · · · · · · · , u_{m}] [ \begin{matrix}σ_{1} \\ & . & . & . \\ & & & & σ_{r} \\ & & & & & 0 \\ & & & & & & . & . & . \\ & & & & & & & & & 0 \end{matrix} ] [ v_{1}^{T} ; ⋮ ; ⋮ ; ⋮ ; v_{n}^{T} ] = ∑_{i=1}^{r} σ_{i}[u_{i}v_{i}^{T}]

(A.183)

by which the original matrix A is represented as a linear combination of r matrices [u_{i}v_{i}^{T}] weighted by the singular values \sqrt{λ_{i}} (i = 1, …, r). We can rewrite both the forward and inverse SVD transform as a pair of forward and inverse transforms:

\{ \begin{matrix}Σ = Λ^{1/2} = U^{T} AV \\ A = UΛ^{1/2}V^{T} = UΣV^{T} . \end{matrix}  (A.184)

This result indicate that the SVD can be used for data compression, if the 2-D data stored in an m × n matrix A can be approximated by keeping in the summation above only k ≪ r ≤ min(m, n) terms corresponding to the k greatest singular values. The amount of data can be significantly reduced from m × n to no more than k(m + n + 1).

Also, the pseudo-inverse of any matrix (non-square, not full-rank) can be found given its SVD

A = UΛ^{1/2}V^{T} = UΣV^{T}  (A.185)

as

A^{−} = (V^{T})^{−1}Σ^{−}U^{−1} = VΣ^{−}U^{T}  (A.186)

where Σ^{−} is pseudo-inverse of Σ, composed of the reciprocals 1/σ_{k} of the R singular values along the diagonal. The pseudo-inverse matrix is needed to find optimal solution of an inconsistent linear equation system.

A.15 Pseudo-Inverse

The inverse of an m × n matrix A does not exist if it is not square m ≠ n. But we can still find its pseudo-inverse, an n × m matrix denoted by A^{−}, if rank(A) = r = min(m, n), in either of the following ways:

• If m > n = r, A^{T} A is an n × n full-rank invertible matrix, and we define the left inverse:

A^{−} = (A^{T} A)^{−1}A^{T}  (A.187)

which satisfies

A^{−}A = (A^{T} A)^{−1}A^{T} A = I_{n×n}  (A.188)

But

AA^{−} = A(A^{T} A)^{−1}A^{T} ≠ I  (A.189)

• If n > m = r, AA^{T} is an m × m full-rank invertible matrix, and we define the right inverse:

A^{−} = A^{T} (AA^{T})^{−1}  (A.190)

which satisfies

AA^{−} = AA^{T} (AA^{T})^{−1} = I_{m×m}  (A.191)

But

A^{−}A = A^{T} (AA^{T})^{−1}A ≠ I  (A.192)

The left and right inverses defined respectively for m > n and m < n are the same. If m > n, then we have the left inverse A^{−} = (A^{T} A)^{−1}A^{T}. We then define B = A^{T}, and take the transpose on A^{−} to get

(A^{−})^{T} = [(A^{T} A)^{−1}A^{T}]^{T} = A(A^{T} A)^{−1} = (A^{T})^{−};  (A.193)

i.e., B^{−} = B^{T} (BB^{T})^{−1}, which is the second definition of the pseudo inverse of B.

We can show that (A^{−})^{−} = A. If m > n, then we have

(A^{−})^{−} = [(A^{T} A)^{−1}A^{T}]^{−} = [(A^{T} A)^{−1}A^{T}]^{T} [(A^{T} A)^{−1}A^{T} [(A^{T} A)^{−1}A^{T}]^{T} ]^{−1}
= A(A^{T} A)^{−1} [(A^{T} A)^{−1}A^{T} A(A^{T} A)^{−1}]^{−1}
= A(A^{T} A)^{−1}(A^{T} A) = A.

Similarly we can show this is the case if m < n.

In particular, when m = n, A is a square invertible matrix, and both the left and right pseudo-inverse defined above become the regular inverse

A^{−} = (A^{T} A)^{−1}A^{T} = A^{−1}(A^{T})^{−1}A^{T} = A^{−1}
A^{−} = A^{T} (AA^{T})^{−1} = A^{T} (A^{T})^{−1}A^{−1} + A^{−1}

If r < min(m, n), then A^{−1} does not exist, and A^{−} is neither left nor right inverse, because A^{−}A ≠ I and AA^{−} ≠ I, as they are not full rank.

The pseudo inverse of A can also be obtained based on its SVD A_{m×n} = U_{m×m}Σ_{m×n}V_{n×n}^{T}. Consider the following two cases:

• If m > n, the left pseudo inverse is

A^{−} = (A^{T} A)^{−1}A^{T} = [(UΣV^{T})^{T} (UΣV^{T})]^{−1}(UΣV^{T})^{T}
= (VΣ^{T} U^{T} UΣV^{T})^{−1}(VΣ^{T} U^{T}) = V(Σ^{T} Σ)^{−1}V^{T} VΣ^{T} U^{T}
= V[(Σ^{T} Σ)^{−1}Σ^{T}]U^{T} = VΣ^{−}U^{T}

• If m < n, the right pseudo inverse is

A^{−} = A^{T} (AA^{T})^{−1} = (UΣV^{T})^{T} [(UΣV^{T})(UΣV^{T})^{T}]^{−1}
= (VΣ^{T} U^{T})(UΣV^{T} VΣ^{T} U^{T})^{−1} = VΣ^{T} U^{T} (UΣΣ^{T} U^{T})^{−1}
= VΣ^{T} U^{T} U(ΣΣ^{T})^{−1}U^{T} = V[Σ^{T} (ΣΣ^{T})^{−1}]U^{T} = VΣ^{−}U^{T}

In either case, we have

A_{n×m}^{−} = (U_{m×m}Σ_{m×m}V_{n×n}^{T})^{−} = V_{n×n}Σ_{n×m}^{−}U_{m×m}^{T}  (A.194)

where Σ_{n×m}^{−} is the pseudo inverse of Σ_{m×n} with 1/σ_{1}, ⋯, 1/σ_{r} on its diagonal. We also have

A^{−}A = (VΣ^{−}U^{T})(UΣV^{T}) = V(Σ^{−}Σ)V^{T}  (A.195)

if r = m, then Σ^{−}Σ = I an n×n identity matrix and we have A^{−}A = VV^{T} = I. Otherwise if r < m, Σ^{−}Σ has zero components on the diagonal, and A^{−}A ≠ I.

A.16 Vector Norms

The norm of a vector x, denoted by ||x||, can be intuititvely interpretated as its “size”. For example, the norm of a real number x ∈ ℝ in the 1-D real space is its absolute value |x|, or its distance to the origin, and the norm of a complex number z = x + jy ∈ ℂ is its modulus |z| = \sqrt{x^{2} + y^{2}}, its Euclidean distance to the origin. Here is the most general definition of a vector norm:

Definition: The norm of a vector x ∈ V in vector space V is a real nonnegative value representing the length or magnitude of the vector. Specifically, the norm of x ∈ V satisfy the following three conditions:

• Positivity:

||x|| ≥ 0, ||x|| = 0 iff x = 0  (A.196)

• Homogeneity:

||ax|| = |a| ||x||, a ∈ ℝ or ℂ  (A.197)

• Triangle inequality:

||x + y|| ≤ ||x|| + ||y||  (A.198)

The triangle inequality can also be expressed in alternative forms. If we define z = −y, the triangle inequality becomes

||x − z|| ≤ ||x|| + ||z||  (A.199)

If we further define u = x − z, i.e., z = x − u, then the above becomes

||u|| ≤ ||x|| + ||x − u||, i.e. ||u|| − ||x|| ≤ ||x − u|| = ||u − x|| (A.200)

Combining the two results above, we get

||x|| − ||y|| ≤ ||x − y|| ≤ ||x|| + ||y||  (A.201)

The first equality holds if x and y are in the same direction, the second equality holds if they are in opposite direction. More specially when y = 0, both equalities hold.

The p-norms of an n-D vector x = [x_{1}, ⋯, x_{n}]^{T} and a function f(x) are defined as:

||x||_{p} = (∑_{i=1}^{n} |x_{i}|^{p})^{1/p} ,  ||f(x)||_{p} = (∫ |f(x)|^{p} dx)^{1/p}  (A.202)

The p-norm satisfies the three requirements in the definition of vector norm. The first two are trivially obvious, while the third one happens to be Minkowski’s inequality (see appendix):

||x + y||_{p} ≤ ||x||_{p} + ||y||_{p}  (A.203)

The p-norms corresponding to p = 1, p = 2, and p = ∞ are most commonly used:

• p = 1, the absolute sum of all n elements:

||x||_{1} = ∑_{i=1}^{n} |x_{i}|  (A.204)

• p = 2, the Euclidean norm:

||x||_{2} = (x^{T} x)^{1/2} = \sqrt{∑_{i=1}^{n} |x_{i}|^{2}}  (A.205)

• p = ∞, the maximum absolute value of all n elements:

||x||_{∞} = (∑_{i=1}^{n} |x_{i}|^{∞})^{1/∞} = max{|x_{1}|, ⋯, |x_{n}|}  (A.206)

Out of the three p-norms, the Euclidean 2-norm is the only one that is unitary invariant, i.e., it is conserved or invariant under any unitary transform y = Rx, where R is a unitary matrix satisfying R^{−1} = R^{∗}:

||y||_{2}^{2} = y^{∗}y = (Rx)^{∗}(Rx) = x^{∗} (R^{∗}R) x = x^{∗}x = ||x||_{2}^{2}  (A.207)

i.e., the length of the vector is not changed by any unitary transform (such as rotation, when R is a rotation matrix).

Definition: Two norms ||x||_{a} and ||x||_{b} are equivalent if there exist two positive real constants c and C so that

c ||x||_{b} ≤ ||x||_{a} ≤ C ||x||_{b}  (A.208)

Note that this relationship can also be written as

\frac{1}{C} ||x||_{a} ≤ ||x||_{b} ≤ \frac{1}{c} ||x||_{a}  (A.209)

Theorem: All different norms x are equivalent. Proof

• We first show that equivalence is transitive, i.e., if both ||x||_{a} and ||x||_{b} are equivalent to ||x||_{1}:

c ||x||_{1} ≤ ||x||_{a} ≤ C ||x||_{1}, and d ||x||_{1} ≤ ||x||_{b} ≤ D ||x||_{1}, (A.210)

then they are equivalent to each other.

From the first equation we get ||x||_{1} ≤ ||x||_{a}/c and ||x||_{1} ≥ ||x||_{a}/C. Substituting these into the right and left hand sides of the second equation respectively yields:

\frac{d}{C} ||x||_{a} ≤ ||x||_{b} ≤ \frac{D}{c} ||x||_{a},  (A.211)

i.e., ||x||_{a} and ||x||_{b} are equivalent.

• If we can further show that an arbitrary norm ||x|| is equivalent to ||x||_{1}, i.e.,

c ||x||_{1} ≤ ||x|| ≤ C ||x||_{1}  (A.212)

then according to the transitivity property shown above, all norms are equivalent.

If x = 0, it is obvious that the equalities above hold ||x|| = ||x||_{1} = 0. If x ≠ 0, ||x||_{1} > 0, we can define a normalized vector u = x/||x||_{1} with ||u||_{1} = 1, so that the relationship for the equivalence above can be written in terms of u as

c = c||u||_{1} ≤ ||u|| ≤ C||u||_{1} = C  (A.213)

i.e., to prove that all norms are equivalent all we need to show is that ||u|| is bounded both from above and below, as we will do below.

• We now prove that an arbitrary norm f(x) = ||x|| is a continuous function over ||x||_{1}, i.e., for any ϵ > 0, there exists a δ > 0 so that

if ||x − x′||_{1} < δ, then ||x|| − ||x′|| ≤ ϵ  (A.214)

Both x and x′ can be expressed in terms of a basis {b_{1}, ⋯, b_{n}} that spans the space:

x = ∑_{i=1}^{n} a_{i}b_{i}, x′ = ∑_{i=1}^{n} a′_{i}b_{i}  (A.215)

then we have

||x−x′|| = || ∑_{i=1}^{n}(a_{i}−a′_{i})b_{i}|| ≤ ∑_{i=1}^{n} |a_{i}−a′_{i}| ||b_{i}|| ≤ ∑_{i=1}^{n} |a_{i}−a′_{i}| max_{i} ||b_{i}|| = ||x−x′||_{1} max_{i} ||b_{i}||

(A.216)

Now consider the alternative form of the triangle inequality of ||x||:

||x|| − ||x′|| ≤ ||x − x′|| ≤ ||x − x′||_{1} max_{i} ||b_{i}||  (A.217)

If we let max_{i} ||b_{i}|| = ϵ/δ, the above becomes

||x|| − ||x′|| ≤ ||x − x′||_{1} \frac{ϵ}{δ}  (A.218)

indicating that for any given ϵ, we can choose δ = ϵ/ max_{i} ||b_{i}|| so that if ||x−x′||_{1} ≤ δ, then ||x||−||x′|| ≤ ϵ, i.e., the norm ||x|| is indeed continuous over ||x||_{1}.

• Finally, according to the extreme value theorem, a continuous function, such as f(x) = ||x||, defined over a compact (closed and bounded) set, such as the unit sphere ||u||_{1} = 1 in the n-D space, must have its maximum and minimum values:

c = min_{||u||_{1}=1} ||u||, C = max_{||u||_{1}=1} ||u||  (A.219)

i.e., c ≤ ||u|| ≤ C, which is what we need to prove.

Q.E.D.

The three p-norms are equivalent:

||x||_{∞} ≤ ||x||_{1} ≤ n||x||_{∞}
||x||_{∞} ≤ ||x||_{2} ≤ \sqrt{n}||x||_{∞}
||x||_{2} ≤ ||x||_{1} ≤ \sqrt{n}||x||_{2}

The distance between two vectors x, y ∈ V is defined as the norm of their difference d(x, y) = ||x − y||. In particular, when y = 0, the distance ||x − 0|| = ||x|| between x and the origin 0 of the space is the norm of x. Specifically, d(x, y) can be defined based on the p-norm:

d_{p}(x, y) = ||x − y||_{p} = (∑_{i=1}^{n} |x_{i} − y_{i}|^{p})^{1/p} , (1 ≤ p ≤ ∞)  (A.220)

Specially, when

• p = 1, d_{1}(x, y) is the city block (Manhattan) distance:

d_{1}(x, y) = ||x − y||_{1} = ∑_{i=1}^{n} |x_{i} − y_{i}|  (A.221)

• p = 2, d_{2}(x, y) is the Euclidean distance:

d_{2}(x, y) = ||x − y||_{2} = \sqrt{∑_{i=1}^{n} |x_{i} − y_{i}|^{2}}  (A.222)

• p = ∞, d_{∞}(x, y) is the Chebyshev distance:

d_{∞}(x, y) = ||x − y||_{∞} = max_{i} |x_{i} − y_{i}|  (A.223)

The three unit “circles” or “spheres”, are formed by all points x of unity norm ||x||_{p} = 1 with unity distance to the origin (blue, black, and red for d_{∞}, d_{2}, and d_{1}, respectively).

A.17 Matrix Norms

The norm of an m × n matrix A is any function that maps A to a real number ||A|| that satisfies the following required properties:

• Positivity:

||A|| ≥ 0, ||A|| = 0 iff A = 0  (A.224)

• Homogeneity:

||aA|| = |a| ||A||  (A.225)

• Triangle inequality:

||A + B|| ≤ ||A|| + ||B||, or ||A|| − ||B|| ≤ ||A − B|| (A.226)

The second expression can be obtained by replacing A by A − B in the first expression.

In addition to the three required properties for matrix norms, some of them also satisfy these additional properties (not necessarily satisfied by other matrix norms):

• Subordinance:

||Ax|| ≤ ||A|| · ||x||  (A.227)

• submultiplicativity:

||AB|| ≤ ||A|| · ||B||  (A.228)

We now consider some commonly used matrix norms:

• The element-wise norms

If we treat the m×n elements of A are the elements of an mn-dimensional vector, then the p-norm of this vector can be used as the p-norm of A:

||A||_{p} = {∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{p}}^{1/p}  (A.229)

Specially, we consider the following three cases for p = 1, 2, ∞.

– ||A||_{1} is the absolute sum of all elements of A:

||A||_{1} = ∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|  (A.230)

– ||A||_{∞} is the maximum norm, the maximum absolute value among all mn elements of A:

||A||_{∞} = max{|a_{ij}|, 1 ≤ i ≤ m, 1 ≤ j ≤ n}  (A.231)

– ||A||_{2} is the Frobenius norm

||A||_{2} = ||A||_{F} = \sqrt{∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{2}} = \sqrt{tr(A^{∗}A)} = \sqrt{∑_{i=1}^{R} λ_{i}} = \sqrt{∑_{i=1}^{R} σ_{i}^{2}}

(A.232)

where r ≤ min(m, n) is the rank of A, λ_{i} = σ_{2}^{2} is the ith non-zero eigenvalues of the positive, semi-definite matrix A^{∗}A, and σ_{i} = \sqrt{λ_{i}} the ith singular value of A (i = 1, ⋯, r). (This Frobenius norm ||A||_{2} is implemented in Matlab by the function norm(A,'fro').)

The Frobenius norm of a unitary (orthogonal if real) matrix R satisfying R^{∗} = R^{−1} or R^{∗}R = R^{−1}R = I is:

||R||_{F}^{2} = tr(R^{∗}R) = trI = n  (A.233)

The Frobenius norm is the only one out of the above three matrix norms that is unitary invariant, i.e., it is conserved or invariant under a unitary transformation (such as a rotation) B = R^{∗}AR:

||B||_{F}^{2} = tr (B^{∗}B) = tr [(R^{∗}AR)^{∗} (R^{∗}AR)] = tr (R^{∗}A^{∗}RR^{∗}AR)
= tr (R^{∗}A^{∗}AR) = tr (A^{∗}A) = ||A||_{F}^{2} = ||A||_{F}^{2}

where we have used the property of the trace tr (AB) = tr (BA).

• The natural matrix norms

The natural norm ||A|| of matrix A induced by (subordinate to) the vector norm ||x|| is defined as

||A|| = sup_{||x||=1}{||Ax||} = sup_{x≠0}{||Ax||/||x||}  (A.234)

Specially the norm of the identity matrix A = I is

||I|| = sup_{||x||=1}{||Ix||} = sup_{||x||=1}{||x||} = 1  (A.235)

We now prove that the induced matrix norm satisfies all five properties given above. (Recall |a + b| ≤ |a| + |b|, |ab| ≤ |a| |b|.) 1.Positivity: ||A|| > 0 if A ≠ 0, this is trivially obvious. 2.Homogeneity: ||aA|| = |a| ||A||

||aA|| = sup_{||x||=1}{||aAx||} = sup_{||x||=1}{|a| ||Ax||} = |a| sup_{||x||=1}{||Ax||} = |a| ||A||

(A.236)

3.Triangle inequality: ||A + B|| ≤ ||A|| + ||B||

||A + B|| = sup_{||x||=1}{||(A + B)x||} = sup_{||x||=1}{||Ax + Bx||} ≤ sup_{||x||=1}{||Ax|| + ||Bx||}
≤ sup_{||x||=1}{||Ax||} + sup_{||x||=1}{||Bx||} = ||A|| + ||B||

4.Subordinance: ||Ax|| ≤ ||A|| ||x||

||A|| ||x|| = ||x|| sup_{y}{||Ay||/||y||} ≥ ||x|| ||Ay||/||y||  (A.237)

As y is arbitrary, we let y = x and get

||A|| ||x|| ≥ ||Ax||  (A.238)

In particular, if vector x is the eigenvector corresponding to the greatest eigenvalue λ_{max} of A^{∗}A:

(A^{∗}A) x = λ_{max} x = ||A||^{2} x  (A.239)

then the equality of the subordinance property holds. To see this, consider

||Ax||^{2} = ⟨(Ax)^{∗}, (Ax)⟩ = x^{∗}(A^{∗}A)x = x^{∗}||A||^{2}x = ||A||^{2}||x||^{2}

(A.240)

taking square root on both sides we get ||Ax|| = ||A|| ||x||. 5.Submultiplicativity: ||AB|| ≤ ||A|| ||B||

||AB|| = sup_{||x||=1}{||ABx||} ≤ sup_{||x||=1}{||A|| ||Bx||} = ||A|| sup_{||x||=1}{||Bx||} = ||A|| ||B||

(A.241)

In particular, if A = cB^{∗} (i.e., A and B are linearly dependent), then the equality of the submultiplicativity property holds. To see this, consider

||B|| = \sqrt{λ_{max}(B^{∗}B)}, ||A|| = |c|\sqrt{λ_{max}(BB^{∗})} = |c|\sqrt{λ_{max}(B^{∗}B)}

(A.242)

and

||AB|| = |c| ||B^{∗}B|| = |c| \sqrt{λ_{max}((B^{∗}B)(B^{∗}B))} = |c| \sqrt{λ_{max}(B^{∗}B)^{2}} = |c| λ_{max}(B^{∗}B)

(A.243)

i.e., ||AB|| = ||A|| ||B||.

Consider in particular the natural matrix norms ||A||_{p} corresponding to the three vector p-norms ||x||_{p} with p = 1, 2, ∞:

– When p = 1, ||A||_{1} (norm(A,1) in Matlab) is the maximum absolute column sum:

||A||_{1} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|  (A.244)

Proof: The 1-norm of vector x is ||x||_{1} = ∑_{j=1}^{n} |x_{j}|, we have

||Ax||_{1} = ∑_{i=1}^{m} | ∑_{j=1}^{n} a_{ij}x_{j}| ≤ ∑_{i=1}^{m} ∑_{j=1}^{n} (|a_{ij}| |x_{j}|) = ∑_{j=1}^{n} (|x_{j}| ∑_{i=1}^{m} |a_{ij}|)

≤ ( max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|) (∑_{j=1}^{n} |x_{j}|) = ||A||_{1} ||x||_{1}

Assuming the kth column of A has the maximum absolute sum and x is normalized (as required in the definition) with ||x|| = 1, we have

||A||_{1} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}| = ∑_{i=1}^{m} |a_{ik}|  (A.245)

and

||Ax||_{1} ≤ ||A||_{1} ||x||_{1} = ∑_{i=1}^{m} |a_{ik}|  (A.246)

Now we show that the equality of the above can be achieved, i.e., ||Ax||_{1} is maximized, if we choose x = e_{k} = [0, ⋯, 1, ⋯, 0]^{T}, the kth unit vector (normalized):

||Ax||_{1} = ∑_{i=1}^{m} | ∑_{j=1}^{n} a_{ij}x_{j}| = ∑_{i=1}^{m} |a_{ik}| = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}| (A.247)

i.e., x = e_{k} is the vector among all other normalized vectors that maximizes ||Ax|| as required in the definition, and the resulting maximum ||A||_{1} is indeed max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|. We therefore have

||A||_{1} = sup_{||x||_{1}=1}{||Ax||_{1}} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|  (A.248)

– When p = ∞, ||A||_{∞} (norm(A,inf) in Matlab) is the maximum absolute row sum:

||A||_{∞} = max_{1≤i≤m} ∑_{j=1}^{n} |a_{ij}|  (A.249)

Proof: When p = ∞, x is normalized if ||x||_{∞} = max{x_{1}, ⋯, x_{n}} = 1. The norm of vector Ax is:

||Ax||_{∞} = max {∑_{j=1}^{n} |a_{ij}x_{j}|, ⋯, ∑_{j=1}^{n} |a_{mj}x_{j}|}  (A.250)

which can be maximized by any normalized vector with ||x||_{∞} = max{x_{1}, ⋯, x_{n}} = 1 to become

||Ax||_{∞} = max {∑_{j=1}^{n} |a_{ij}|, ⋯, ∑_{j=1}^{n} |a_{mj}|}  (A.251)

We therefore have

||A||_{∞} = sup_{||x||_{∞}=1}{||Ax||_{∞}} = max_{i} ∑_{j=1}^{n} |a_{ij}|  (A.252)

For example, for any stochastic matrix P satisfying ∑_{j} P_{ij} = 1, ||P||_{∞} = 1.

– When p = 2, ||A||_{2} (norm(A,2) or norm(A) in Matlab), also called the spectral norm, is the greatest singular value of A, square root of the greatest eigenvalue of A^{∗}A, i.e., its spectral radius ρ(A^{∗}A) = λ_{max}(A^{∗}A):

||A||_{2} = σ_{max}(A) = \sqrt{λ_{max}(A^{∗}A)} = \sqrt{ρ(A^{∗}A)}  (A.253)

where λ_{max}(A^{∗}A) = σ_{max}^{2}(A) is the maximal eigenvalue of A^{∗}A. Proof: First we consider the eigenequation of A^{∗}A:

A^{∗}Av = λ_{i}v_{i}, (i = 1, ⋯, n) or A^{∗}AV = VΛ, or A^{∗}A = VΛV^{∗}

(A.254)

where

Λ = diag(λ_{1}, ⋯, λ_{n}), and V = [v_{1}, ⋯, v_{n}]  (A.255)

are the eigenvector and eigenvalue matrices of A^{∗}A. As A^{∗}A is symmetric and positive definite, all of its eigenvalues are real and positive, and assumed to be sorted

λ_{1} = λ_{max} ≥ λ_{2} ≥ ⋯ ≥ λ_{n} = λ_{min} ≥ 0  (A.256)

and all corresponding eigenvectors are orthogonal and assumed to be normalized, i.e., v_{i}^{∗}v_{j} = δ_{ij}, or V^{∗} = V^{−1} is a unitary (orthogonal if real) matrix. Next, consider the p = 2 norm of Ax:

||Ax||_{2} = [(Ax)^{∗}(Ax)]^{1/2} = (x^{∗}A^{∗}Ax)^{1/2} = (x^{∗}VΛV^{∗}x)^{1/2}
= (y^{∗}Λy)^{1/2} = \sqrt{∑_{i=1}^{n} y_{i}^{2}λ_{i}}

Here y = V^{∗}x is a unitary transform of x with the 2-norm ||y||_{2} = ||x||_{2} conserved. The right-hand side of the equation above is a weighted average of the n eigenvalues λ_{1}, ⋯, λ_{n}, which is maximized if they are weighted by a normalized vector y = [1, 0, ⋯, 0]^{T} with ||y||_{2} = 1, by which the greatest eigenvalue λ_{max} is maximally weighted while all others are weighted by 0:

||Ax||_{2} ≤ \sqrt{λ_{max}}  (A.257)

As ||x||_{2} = ||y||_{2} = 1, we have

||A||_{2} = sup_{||x||_{2}=1}\{||Ax||_{2}\} = \sqrt{λ_{max}}  (A.258)

Out of the three matrix norms defined above, the spectral norm is the only one that is unitary invariant, i.e., it is conserved or invariant under any unitary transform B = R^{∗}AR:

||B||_{2}^{2} = λ_{max} (B^{∗}B) = λ_{max} [(R^{∗}AR)^{∗} (R^{∗}AR)] = λ_{max} (R^{∗}A^{∗}RR^{∗}AR)
= λ_{max} (R^{∗}A^{∗}AR) = λ_{max} (A^{∗}A) = ||A||_{2}^{2}

Here we have used the fact that the eigenvalues and eigenvectors are invariant under the unitary transform.

Example

A = [ \begin{matrix}3 & −6 & 2 \\ 2 & 5 & 1 \\ −3 & 2 & 2 \end{matrix} ] ,  B = [ \begin{matrix}3 & 2 & 1 \\ 2 & −3 & 0 \\ 1 & 0 & −1 \end{matrix} ]  (A.259)

||A||_{1} = max{|3| + |2| + | − 3|, | − 6| + |5| + |2|, |2| + |1| + |2|} = max{8, 13, 5} = 13
||A||_{∞} = max{|3| + | − 6| + |2|, |2| + |5| + |1|, | − 3| + |2| + |2|} = max{11, 8, 7} = 11

The eigenvalues of A^{∗}A are

λ_{1} = 69.353, λ_{2} = 17.967, λ_{3} = 8.680  (A.260)

The singular values of A are

σ_{1} = \sqrt{λ_{1}} = 8.328, σ_{2} = \sqrt{λ_{2}} = 4.239, σ_{3} = \sqrt{λ_{3}} = 2.946 (A.261)

The spectral norm of A is

||A||_{2} = σ_{max} = \sqrt{λ_{max}} = 8.328  (A.262)

The eigenvector corresponding to greatest eigenvalue λ_{1} is x = [0.285, −0.957, 0.057]^{T}, which satisfies the equality ||Ax|| = ||A|| ||x||.

• The Schatten norms

The Shatten norm is defined based on the singular values σ_{i} of A or the eigenvalues λ_{i} = σ^{2} of A^{∗}A:

||A||_{p} = (∑_{i=1}^{R} σ_{i}^{p})^{1/p} = (∑_{i=1}^{R} (\sqrt{λ_{i}})^{p})^{1/p}  (A.263)

In particular, consider three common p values:

– p = 1 is the nuclear or trace norm:

||A||_{1} = ∑_{i=1}^{R} σ_{i} = ∑_{i=1}^{R} \sqrt{λ_{i}} = tr (\sqrt{A^{∗}A})  (A.264)

– p = 2 same as the Frobenius norm:

||A||_{2} = \sqrt{∑_{i=1}^{R} σ_{i}^{2}} = \sqrt{∑_{i=1}^{R} λ_{i}} = ||A||_{F}  (A.265)

– p = ∞ same as the spectral norm (the induced 2-norm), the spectral radius of A^{∗}A.

||A||_{∞} = max\{σ_{1}, ⋯, σ_{n}\} = σ_{max} = \sqrt{λ_{max}}  (A.266)

As the eigenvalues and eigenvectors of A^{∗}A are invariant under unitary transform, the Schatten norms are unitary invariant as well.

All matrix norms defined above are equivalent according to the theorem previously discussed.

• The Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{R} λ_{i}} and the induced 2-norm ||A||_{2} = \sqrt{λ_{max}} are equivalent:

||A||_{2} ≤ ||A||_{F} ≤ \sqrt{R} ||A||_{2}  (A.267)

The equality on the left holds when all eigenvalues λ_{i} but one are zero, and the equality on the right holds when all λ_{i} are the same.

• The Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{R} λ_{i}} and the Schatten 1-norm ||A||_{1} = ∑_{i=1}^{R} \sqrt{λ_{i}} are equivalent:

||A||_{F} ≤ ||A||_{1} ≤ \sqrt{R} ||A||_{F}  (A.268)

The equality on the left holds when all eigenvalues λ_{i} but one are zero, and the equality on the right holds when all λ_{i} are the same.

• The element-wise maximum norm ||A||_{max} = max_{i,j}{|a_{ij}|} and the Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{2}} are equivalent:

||A||_{max} ≤ ||A||_{F} ≤ \sqrt{mn} ||A||_{max}  (A.269)

The equality on the left holds when all elements λ_{i} but one are zero, and the equality on the right holds when all elements are the same.

•

\frac{1}{\sqrt{n}}||A||_{∞} ≤ ||A||_{2} ≤ \sqrt{m}||A||_{∞}  (A.270)

Proof: Define an n-D vector e = [1, ⋯, 1]^{T}, then the greatest absolute row sum of A is

||A||_{∞} = max_{i} ∑_{j=1}^{n} |a_{ij}| = ||Ae||_{∞} ≤ ||Ae||_{2} ≤ ||A||_{2} ||e||_{2} = \sqrt{n}||A||_{2}

(A.271)

i.e.,

\frac{1}{\sqrt{n}}||A||_{∞} ≤ ||A||_{2}  (A.272)

•

\frac{1}{\sqrt{m}}||A||_{1} ≤ ||A||_{2} ≤ \sqrt{n}||A||_{1}  (A.273)

Theorem: The spectral radius of a matrix is bounded by its matrix norm: ρ(A) ≤ ||A||

Proof: Let λ and u be an eigenvalue and the corresponding normalized eigenvector of a square matrix A, i.e., Au = λu and ||u|| = 1. As the induced norm, ||A|| satisfies

||A|| = sup_{||x||=1}{||Ax|} ≥ ||Au|| = ||λu|| = |λ| ||u|| = |λ|  (A.274)

But λ is an arbitrary eigenvalue, it can be the one with the maximum absolute value |λ|, i.e., the spectral radius ρ(A). We therefore have ||A|| ≥ ρ(A).

Theorem:

ρ(A) ≤ ||A^{k}||^{1/k}  (A.275)

Proof: Let λ and v by the eigenvalue and the corresponding eigenvector of A respectively, i.e.,

Av = λv, and A^{k}v = λ^{k}v  (A.276)

Taking norm on both sides we get

||λ^{k}v|| = |λ|^{k}||v|| = ||A^{k}v|| ≤ ||A^{k}|| ||v||  (A.277)

Dividing both sides by ||v|| ≠ 0 we get

λ^{k} ≤ ||A^{k}||, i.e. λ ≤ ||A^{k}||^{1/k}  (A.278)

Theorem: A square matrix A is convergent, i.e., lim_{n→∞} A^{n} = 0, if and only if ρ(A) < 1.

  The proof of this theorem can be found at http://en.wikipedia.org/wiki/Spectral_radius.

A.18 Vector and Matrix Differentiation

A multivariate function f(x) = f(x_{1}, ⋯, x_{n}) can be considered as a hypersurface in an n + 1 dimensional space defined over an n-D space, i.e., corresponding to every point x = [x_{1}, ⋯, x_{n}]^{T} in the n-D space, there exists a unique value f(x) (real or complex).

A vector differentiation operator is defined as

\frac{d}{dx} =^{△} [\frac{∂}{∂x_{1}} , ⋯, \frac{∂}{∂x_{n}} ]^{T}  (A.279)

which, when applied to any scalar function f(x), produces its derivative with respect to x, the gradient vector of f(x):

g_{f}(x) = ▽f(x) = \frac{d}{dx}f(x) = [\frac{∂f}{∂x_{1}} , ⋯, \frac{∂f}{∂x_{n}} ]^{T} = [f_{x_{1}}, ⋯, f_{x_{n}}]^{T} (A.280)

The Directional derivative of a function f(x) is defined as the rate of change of the function value along a unit direction r = [r_{1}, ⋯, r_{n}]^{T} (with ||r|| = 1):

D_{r}f(x) = ▽_{r}f(x) = \frac{d}{dh}f(x + hr) = lim_{h→0} \frac{f(x + hr) − f(x)}{h}  (A.281)

Introducing n intermediate functions x_{i} +hr_{i} (i = 1, ⋯, n) of h, we can rewrite the above as the following by chain rule:

D_{r}f(x) = \frac{d}{dh}f(x+hr)|_{h=0} = ∑_{i=1}^{n} \frac{∂f(x + hr)}{∂(x_{i} + hr_{i})} \frac{d(x_{i} + hr_{i})}{dh}|_{h=0} = ∑_{i=1}^{n} f_{x_{i}} r_{i} = g_{f}^{T} r

(A.282)

We see that the directional derivative is the projection of the gradient vector onto the unit direction r, which is maximized when the gradient vector g_{f} is in the same direction as r. In other words, the function value increases mot rapidly along the direction of its gradient.

Vector differentiation has the following properties:

\frac{d}{dx}(a^{T} x) = \frac{d}{dx}(x^{T} a) = a,  \frac{d}{dx}(x^{T} x) = 2x,  \frac{d}{dx}(x^{T} Ax) = (A^{T} + A)x

(A.283)

To prove the third one, consider the kth element of the vector:

\frac{∂}{∂x_{k}} (x^{T} Ax) = \frac{∂}{∂x_{k}} ∑_{i=1}^{n} ∑_{j=1}^{n} a_{ij}x_{i}x_{j} = ∑_{i=1}^{n} a_{ik}x_{i} + ∑_{j=1}^{n} a_{kj}x_{j} (k = 1, ⋯, n)

(A.284)

Putting all n elements in vector form, we have the above. If A^{T} = A is symmetric, then we have

\frac{d}{dx}(x^{T} Ax) = 2Ax  (A.285)

In particular, when A = I, we have

\frac{d}{dx}(x^{T} x) = 2x  (A.286)

One can compare these results with the familiar derivatives in scalar case:

\frac{d}{dx}(ax^{2}) = 2ax  (A.287)

A matrix differentiation operator is defined as

\frac{d}{dA} =^{△} [ \begin{matrix}\frac{∂}{∂a_{11}} & ⋯ & \frac{∂}{∂a_{1n}} \\ ⋮ & ... & ⋮ \\ \frac{∂}{∂a_{m1}} & ⋯ & \frac{∂}{∂a_{mn}} \end{matrix} ]  (A.288)

which can be applied to any scalar function f(A):

\frac{d}{dA}f(A) = [ \begin{matrix}\frac{∂f(A)}{∂a_{11}} & ⋯ & \frac{∂f(A)}{∂a_{1n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f(A)}{∂a_{m1}} & ⋯ & \frac{∂f(A)}{∂a_{mn}} \end{matrix} ]  (A.289)

Specifically, consider f(A) = u^{T} Av, where u and v are m × 1 and n × 1 constant vectors, respectively, and A is an m × n matrix. Then we have:

\frac{d}{dA}(u^{T} Av) = uv^{T}  (A.290)

For more details refer to Matrix Cookbook at: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf.

A.19 The Fundamental Theorem of Linear Algebra

When solving a linear equation system Ax = b with an M ×N coefficient matrix A, we need to answer some questions such as the following:

• Does a solution exist, i.e., can we find an x^{∗} so that Ax^{∗} = b holds?

• If such a solution exists, is it unique? If it is not unique, how can we find all of the solutions?

• If no solution exists, can we still find the optimal approximated solution x̂ so that the error ||e|| = ||Ax̂ − b|| is minimized?

• If the system has fewer equations than unknowns (M < N), are there infinite solutions?

• If the system has more equations than unknowns (M > N), is there no solution?

To answer all such questions, we consider the fundamental theorem of linear algebra to reveal the structure of the solutions of a given linear system Ax = b.

The M × N coefficient matrix A ∈ R^{M×N} can be expressed in terms of either its N M-D column vectors c_{j} or its M N-D row vectors r_{i}^{T}:

A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ] = [c_{1}, ⋯, c_{N}] = [ r_{1}^{T} ; ⋮ ; r_{M}^{T} ] ,  (A.291)

A^{T} = [ \begin{matrix}a_{11} & ⋯ & a_{M1} \\ ⋮ & ... & ⋮ \\ a_{1N} & ⋯ & a_{MN} \end{matrix} ] = [r_{1}, ⋯, r_{M}] = [ c_{1}^{T} ; ⋮ ; c_{N}^{T} ]  (A.292)

where the ith row r_{i}, (i = 1, ⋯, M) and the jth column c_{j}, (j = 1, ⋯, N) of A are both represented as column vectors:

c_{j} = [ a_{1j} ; ⋮ ; a_{Mj} ] ,  r_{i} = [ a_{i1} ; ⋮ ; a_{iN} ] i.e. r_{i}^{T} = [a_{i1}, ⋯, a_{iN}]  (A.293)

In general a function y = f(x) can be represented by f : X → Y, where

• X is the domain of the function, the set of all input or argument values;

• Y is the codomain of the function, the set into which all outputs of the function fall;

• The image of f(x) is the set formed by the outputs of the function corresponding to all possible x ∈ X, a subset of the codomain.

A linear function f(x) = Ax maps an N-D vector x ∈ ℝ^{N} in the domain of the function into an M-D vector Ax ∈ R^{M} in the codomain of the function. The fundamental theorem of linear algebra concerns the following four subspaces all associated with with the M × N coefficient matrix A with rank R = rank(A) ≤ min(M, N), i.e., A has R independent columns and rows.

• The column space of A is a space spanned by its N M-D column vectors (of which R ≤ N are independent):

C(A) = span(c_{1}, ⋯, c_{N}) ⊆ R^{M}  (A.294)

This is an R-D subspace of R^{M} composed of all possible linear combinations of its N column vectors:

x_{1}c_{1} + ⋯ + x_{N}c_{N} = [c_{1}, ⋯, c_{N}] [ x_{1} ; ⋮ ; x_{N} ] = Ax (for any x ∈ ℝ^{N})

(A.295)

The column space C(A) is the image of the linear transformation f(x) = Ax, and the equation Ax = b is solvable if and only if b ∈ C(A). The dimension of the column space is the rank of A, dim C(A) = rank(A) = R.

• The row space of A is a space spanned by its M N-D row vectors (of which R ≤ M are independent):

R(A) = span(r_{1}, ⋯, r_{M}) ⊆ ℝ^{N}  (A.296)

This is an R-D subspace of ℝ^{N} composed of all possible linear combinations of its M row vectors:

y_{1}r_{1} + ⋯ + y_{M}r_{N} = [r_{1}, ⋯, r_{M}] [ y_{1} ; ⋮ ; y_{M} ] = A^{T} y (for any y ∈ ℝ^{M})

(A.297)

The row space R(A) is the image of the linear transformation f(y) = A^{T} y, and the equation A^{T} y = c is solvable if and only if c ∈ R(A). As the rows and columns in A are respectively the columns and rows in A^{T}, the row space of A is the column space of A^{T}, and the column space of A is the row space of A^{T}:

R(A) = C(A^{T}),  C(A) = R(A^{T})  (A.298)

The rank R = rank(A) is the number of linearly independent rows and columns of A, i.e., the row space and the column space have the same dimension, both equal to the rank of A:

dim C(A) = dim R(A) = rank(A) = R  (A.299)

• The null space (kernel) of A, denoted by N(A), is the set of all N-D vectors x that satisfy the homogeneous equation

Ax = [ r_{1}^{T} ; ⋮ ; r_{M}^{T} ] x = 0, or r_{i}^{T} x = 0 (i = 1, ⋯, M)  (A.300)

i.e.,

N(A) = {x ∈ ℝ^{N}|Ax = 0} ⊆ ℝ^{N}  (A.301)

In particular, when x = 0, we get Ax = 0, i.e., the origin 0 ∈ N(A) is in the null space.

As r_{i}^{T} x = 0 for any x ∈ N(A) and r_{i} ∈ R(A), we see that the null space N(A) and the row space R(A) = C(A^{T}) are orthogonal to each other, N(A) ⊥ R(A).

The dimension of the null space is called the nullity of A: dim N(A) = nullity(A). The rank-nullity theorem states the sum of the rank and the nullity of an M × N matrix A is equal to N:

rank(A) + nullity(A) = dim R(A) + dim N(A) = N  (A.302)

We therefore see that R(A) and N(A) are two mutually exclusive and complementary subspaces of ℝ^{N}:

R(A) ⊥ N(A), R(A) ∩ N(A) = ∅, R(A) ⊕ N(A) = ℝ^{N} (A.303)

i.e., they are orthogonal complement of each other, denoted by

N(A) = R(A)^{⊥}, R(A) = N(A)^{⊥}  (A.304)

Any N-D vector x ∈ ℝ^{N} is in either of the two subspaces R(A) and N(A).

• The null space of A^{T} (left null space of A), denoted by N(A^{T}), is the set of all M-D vectors y that satisfy the homogeneous equation

A^{T} y = [c_{1}, ⋯, c_{N}]^{T} y = [ c_{1}^{T} ; ⋮ ; c_{N}^{T} ] y = 0, or c_{j}^{T} y = 0, (j = 1, ⋯, N)

(A.305)

i.e.,

N(A^{T}) = {y ∈ ℝ^{M}|A^{T} y = 0} ⊆ ℝ^{M}  (A.306)

As all y ∈ N(A^{T}) are orthogonal to c_{i} ∈ C(A), N(A^{T}) is orthogonal to the column space C(A) = R(A^{T}):

N(A^{T}) ⊥ C(A),  N(A^{T}) ⊥ R(A^{T})  (A.307)

We see that C(A) and N(A^{T}) are two mutually exclusive and complementary subspaces of ℝ^{M}:

C(A) ⊥ N(A^{T}), C(A) ∩ N(A^{T}) = ∅, C(A) ⊕ N(A^{T}) = R^{M},

i.e. N(A^{T}) = C(A)^{⊥}, C(A) = N(A^{T})^{⊥}  (A.308)

Any M-D vector y ∈ ℝ^{M} is in either of the two subspaces C(A) and N(A^{T}).

The four subspaces are summarized in the figure below, showing the domain ℝ^{N} = R(A) ⊕ N(A) (left) and the codomain ℝ^{M} = C(A) ⊕ N(A^{T}) (right) of the linear mapping Ax = b, where

• x_{p} ∈ R(A) is the particular solution that is mapped to Ax_{p} = b ∈ C(A), the image of f(x) = Ax;

• x_{h} ∈ N(A) is a homogeneous solution that is mapped to Ax_{h} = 0 ∈ ℝ^{M};

• x_{c} = x_{p} + x_{h} is the complete solution that is mapped to Ax_{c} = b ∈ C(A).

On the other hand, y_{p} ∈ C(A) = R(A^{T}), y_{h} ∈ N(A^{T}), and y_{c} = y_{p} + y_{h} are respectively the particular, homogeneous and complete solutions of A^{T} y = c. Here we have assumed b ∈ C(A) and c ∈ R(A), i.e., both Ax = b and A^{T} y = c are solvable. We will also consider the case where b ∉ C(A) later.

We now consider specifically how to find the solutions of the system Ax = b in light of the four subspaces of A defined above, through the examples below.

Example 1:

Solve the homogeneous equation system:

Ax = [ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 0 \end{matrix} ] = 0  (A.309)

We first convert A into the rref:

[ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 \\ 0 & −1 & −2 \\ 0 & −5 & −10 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] → [ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 0 \end{matrix} ]

(A.310)

The R = rank(A) = 2 columns in the rref containing a single 1, called a pivot, are called the pivot columns, and the rows containing a pivot are called the pivot rows. Here, R = 2 < M = N = 3, i.e., A is a singular matrix. The two pivot rows r_{1}^{T} = [1, 0, 1] and r_{2}^{T} = [0, 1, 2] can be used as the basis vectors that span the row space R(A):

R(A) = c_{1}r_{1} + c_{2}r_{2} = c_{1} [ \begin{matrix}1 \\ 0 \\ 1 \end{matrix} ] + c_{2} [ \begin{matrix}0 \\ 1 \\ 2 \end{matrix} ]  (A.311)

Note that the pivot columns of the rref do not span the column space C(A), as the row reduction operations do not reserve the columns of A. But they indicate the corresponding columns c_{1} = [1 2 3]^{T} and c_{2} = [2 3 1]^{T} in the original matrix A can be used as the basis that spans C(A). In general the bases of the row and column spaces so obtained are not orthogonal.

The R pivot rows are the independent equations in the system of M equations, and the variables corresponding to the pivot columns (here x_{1} and x_{2}) are the pivot variables. The remaining M − R non-pivot rows containing all zeros are not independent, and the variables corresponding to the non-pivot rows are free variables (here x_{3}), which can take any values.

From the rref form of the equation, we get

x_{1} + x_{3} = 0, x_{2} + 2x_{3} = 0  (A.312)

If we let the free variable x_{3} take the value 1, then we can get the two pivot variables x_{1} = −1 and x_{2} = −2, and a special homogeneous solution x_{h} = [−1 − 2 1]^{T} as a basis vector that spans the 1-D null space N(A). However, as the free variable x_{3} can take any value c, the complete solution is the entire 1-D null space:

N(A) = cx_{h} = c [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ]  (A.313)

Example 2:

Solve the non-homogeneous equation with the same coefficient matrix A used in the previous example:

Ax = [ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}19 \\ 31 \\ 22 \end{matrix} ] = b  (A.314)

We use Gauss-Jordan elimination to solve this system:

[ A b ] = [ \begin{matrix}1 & 2 & 5 & 19 \\ 2 & 3 & 8 & 31 \\ 3 & 1 & 5 & 22 \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 & 19 \\ 0 & −1 & −2 & −7 \\ 0 & −5 & −10 & −35 \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 & 19 \\ 0 & 1 & 2 & 7 \\ 0 & 0 & 0 & 0 \end{matrix} ] → [ \begin{matrix}1 & 0 & 1 & 5 \\ 0 & 1 & 2 & 7 \\ 0 & 0 & 0 & 0 \end{matrix} ]

(A.315)

The R = 2 pivot rows correspond to the independent equations in the system, i.e., rand(A) = 2, while the remaining M − R = 1 non-pivot row does not play any role as they map any x to 0. As A is singular, A^{−1} does not exist. However, we can find the solution based on the rref of the system, which can also be expressed in block matrix form:

[ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ]  or  [ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}x_{pivot} \\ x_{free} \end{matrix} ] = [ \begin{matrix}b_{1} \\ 0 \end{matrix} ]

(A.316)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}1 \\ 2 \end{matrix} ] , x_{pivot} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , x_{free} = x_{3}, b_{1} = [ \begin{matrix}5 \\ 7 \end{matrix} ]

(A.317)

Solving the matrix equation above for x_{pivot}, we get

Ix_{pivot}+Fx_{free} = b_{1}, x_{pivot} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = b_{1}−Fx_{free} = [ \begin{matrix}5 \\ 7 \end{matrix} ]+x_{3} [ \begin{matrix}−1 \\ −2 \end{matrix} ]

(A.318)

If we let x_{free} = x_{3} = 0, we get a particular solution x_{p} = [5 7 0]^{T}, which can be expressed as a linear combination of r_{1} and r_{2} that span R(A), and x_{h} that span N(A):

x_{p} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] = \frac{11}{6}r_{1}+ \frac{2}{3}r_{2}− \frac{19}{6}x_{h} = \frac{11}{6} [ \begin{matrix}1 \\ 0 \\ 1 \end{matrix} ]+ \frac{2}{3} [ \begin{matrix}0 \\ 1 \\ 2 \end{matrix} ]− \frac{19}{6} [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ] (A.319)

We see that this solution is not entirely in the row space R(A). In general, this is the case for all particular solutions so obtained.

Having found both the particular solution x_{p} and the homogeneous solution x_{h}, we can further find the complete solution x_{c} as the sum of x_{p} and the entire null space spanned by x_{h}:

x_{c} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] + c [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ] = x_{p} + cx_{h} = x_{p} + N(A)  (A.320)

Based on different constant c, we get a set of equally valid solutions. For example, if c = 0, 1, 2, then we get

x_{0} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] , x_{1} = [ \begin{matrix}4 \\ 5 \\ 1 \end{matrix} ] , x_{2} = [ \begin{matrix}3 \\ 3 \\ 2 \end{matrix} ]  (A.321)

These solutions have the same projection onto the row space R(A), i.e., they have the same projections onto the two basis vectors r_{1} and r_{2} that span R(A):

x_{0}^{T} r_{1} = x_{1}^{T} r_{1} = x_{2}^{T} r_{1} = 5,  x_{0}^{T} r_{2} = x_{1}^{T} r_{2} = x_{2}^{T} r_{2} = 7  (A.322)

The figure below shows how the complete solution x_{c} can be obtained as the sum of a particular solution x_{p} in R(A) and the entire null space N(A). Here N = 3 and space ℝ^{3} is composed of R(A) and N(A), respectively 2-D and 1-D on the left, but 1-D and 2-D on the right. In either case, the complete solution is any particular solution plus the entire null space, the vertical dashed line on the left, the top dashed plane on the right. All points on the vertical line or top satisfy the equation system, as they are have the same projection onto the row space R(A).

If the right hand side is b′ = [19, 31, 20]^{T}, then the rref of the equation becomes:

[ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}5 \\ 7 \\ 2 \end{matrix} ]  (A.323)

The non-pivot row is an impossible equation 0 = 2, indicating that no solution exists, as b′ ∉ C(A) is not in the column space spanned by r_{1} = [1 0 1]^{T} and r_{2} = [0 1 2]^{T}.

Example 3: Find the complete solution of the following linear equation system:

Ax = [ \begin{matrix}1 & 2 & 3 & 4 \\ 4 & 3 & 2 & 1 \\ −2 & 1 & 4 & 7 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = b  (A.324)

This equation can be solved in the following steps:

• Construct the augmented matrix and then convert it to the rref form:

[ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 4 & 3 & 2 & 1 & 2 \\ −2 & 1 & 4 & 7 & 4 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & −5 & −10 & −15 & −10 \\ 0 & 5 & 10 & 15 & 10 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & 1 & 2 & 3 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ]

→ [ \begin{matrix}1 & 0 & −1 & −2 & −1 \\ 0 & 1 & 2 & 3 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ] = [ \begin{matrix}I & F & b_{1} \\ 0 & 0 & 0 \end{matrix} ]

The two pivot rows r_{1}^{T} = [1 0 − 1 − 2] and r_{2}^{T} = [0 1 2 3] in the rref span R(A), and the two columns in the original matrix A corresponding to the pivot columns in the rref, [1 4 − 2]^{T} and [2 3 1]^{T} could be used as the basis vectors that span C(A).

The equation system can be represented in block matrix form:

[ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}x_{p} \\ x_{f} \end{matrix} ] = [ \begin{matrix}b_{1} \\ 0 \end{matrix} ]  (A.325)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}−1 & −2 \\ 2 & 3 \end{matrix} ] , x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , x_{f} = [ \begin{matrix}x_{3} \\ x_{4} \end{matrix} ] , b_{1} = [ \begin{matrix}−1 \\ 2 \end{matrix} ]

(A.326)

Multiplying out we get

x_{p} + Fx_{f} = b_{1}  (A.327)

• Find the homogeneous solution for equation Ax = 0 by setting b_{1} = 0:

Ix_{p} + Fx_{f} = 0, i.e. x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = −Fx_{f} = [ \begin{matrix}1 & 2 \\ −2 & −3 \end{matrix} ] [ \begin{matrix}x_{3} \\ x_{4} \end{matrix} ]

(A.328)

We let x_{f} be either of the two standard basis vectors x_{f} = [x_{3}, x_{4}]^{T} = [1 0]^{T} and x_{f} = [x_{3}, x_{4}]^{T} = [0 1]^{T} of the null space N(A), and get

x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ −2 \end{matrix} ] or [ \begin{matrix}2 \\ −3 \end{matrix} ]  (A.329)

and the two corresponding homogeneous solutions:

x_{h1} = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] ,  x_{h2} = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ] (A.330)

• Find the particular solution of the non-homogeneous equation Ax = b by setting x_{f} = 0

Ix_{p} + Fx_{f} = x_{p} = b_{1}, i.e. [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−1 \\ 2 \end{matrix} ] i.e. x_{p} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ]

(A.331)

• Find the complete solution:

x_{c} = x_{p} +N(A) = x_{p} +c_{1}x_{h1} +c_{2}x_{h2} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ]+c_{1} [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ]+c_{2} [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ]

(A.332)

If the right-hand side is b′ = [1, 3, 5]^{T}, then the row reduction of the augmented matrix yields:

[ \begin{matrix}1 & 2 & 3 & 4 & 1 \\ 4 & 3 & 2 & 1 & 3 \\ −2 & 1 & 4 & 7 & 5 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & −5 & −10 & −15 & −1 \\ 0 & 5 & 10 & 15 & 7 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 1 \\ 0 & 1 & 2 & 3 & 0.2 \\ 0 & 0 & 0 & 0 & 1.2 \end{matrix} ]

(A.333)

The equation corresponding to the last non-pivot row is 0 = 1.2, indicating the system is not solvable (even though the coefficient matrix does not have full rank), because b′ ∉ C(A) is not in the column space.

Example 4: Consider the linear equation system with a coefficient matrix A^{T}, the transpose of A used in the previous example:

A^{T} y = [ \begin{matrix}1 & 4 & −2 \\ 2 & 3 & 1 \\ 3 & 2 & 4 \\ 4 & 1 & 7 \end{matrix} ] [ \begin{matrix}y_{1} \\ y_{2} \\ y_{3} \end{matrix} ] = [ \begin{matrix}3 \\ 11 \\ 19 \\ 27 \end{matrix} ] = c  (A.334)

• Convert the augmented matrix into the rref form:

[ \begin{matrix}1 & 4 & −2 & 3 \\ 2 & 3 & 1 & 11 \\ 3 & 2 & 4 & 19 \\ 4 & 1 & 7 & 27 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 3 \\ 0 & −5 & 5 & 5 \\ 0 & −10 & 10 & 10 \\ 0 & −15 & 15 & 15 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 3 \\ 0 & 1 & −1 & −1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ] → [ \begin{matrix}1 & 0 & 2 & 7 \\ 0 & 1 & −1 & −1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ]

(A.335)

The two pivot rows r_{1}^{T} = [1 0 2] and r_{2}^{T} = [0 1 −1] are the basis vectors that span R(A^{T}) = C(A). The two vectors that span C(A) found in Example 3 can be expressed as linear combinations of r_{1} and r_{2}, [1 4 − 2]^{T} = r_{1} + 4r_{2} and [2 3 1]^{T} = 2r_{1} + 3r_{2}, i.e., either of the two pairs can be used as the basis that span C(A).

Based on the rref above, the equation system can now be written as

[ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}y_{p} \\ y_{f} \end{matrix} ] = [ \begin{matrix}c_{1} \\ 0 \end{matrix} ]  (A.336)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}2 \\ −1 \end{matrix} ] , y_{p} = [ \begin{matrix}y_{1} \\ y_{2} \end{matrix} ] , y_{f} = y_{3}, c_{1} = [ \begin{matrix}7 \\ −1 \end{matrix} ]

(A.337)

Multiplying out we get

Iy_{p} + Fy_{f} = c_{1}  (A.338)

• Find the homogeneous solution for A^{T} y = c = 0 by setting c = 0 and thereby c_{1} = 0. We let the free variable y_{f} = y_{3} = 1 and get

Iy_{p}+Fy_{f} = 0, i.e. y_{p} = −Fy_{f} = [ \begin{matrix}y_{1} \\ y_{2} \end{matrix} ] = − [ \begin{matrix}2 \\ −1 \end{matrix} ] y_{3} = [ \begin{matrix}−2 \\ 1 \end{matrix} ] , i.e. y_{h} = [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]

(A.339)

• Find the particular solution of the non-homogeneous equation A^{T} y = c ≠ 0 by setting y_{f} = y_{3} = 0:

Iy_{p} + Fy_{f} = y_{p} = c_{1} = [ \begin{matrix}7 \\ −1 \end{matrix} ] , i.e. y_{p} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ]  (A.340)

• Find the complete solution:

y_{c} = y_{p} + N(A^{T}) = y_{p} + cy_{3} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ] + c [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]  (A.341)

If the right-hand side is c′ = [1, 1, 2, 3]^{T},

[ \begin{matrix}1 & 4 & −2 & 1 \\ 2 & 3 & 1 & 1 \\ 3 & 2 & 4 & 2 \\ 4 & 1 & 7 & 3 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 1 \\ 0 & −5 & 5 & −1 \\ 0 & −10 & 10 & −1 \\ 0 & −15 & 15 & −1 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 1 \\ 0 & 1 & −1 & 1/5 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 2 \end{matrix} ]

(A.342)

indicating the system is not solvable, as this c′ ∉ C(A^{T}) = R(A), i.e., it is not in the column space of A^{T} or row space of A.

In the two examples above, we have obtained all four subspaces associated with this matrix A with M = 3, N = 4, and R = 2, in terms of the bases that span the subspaces:

1. The row space R(A) = C(A^{T}) is an R-D subspace of ℝ^{4}, spanned by the R = 2 pivot rows of the rref of A:

R(A) = C(A^{T}) = span ([ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ] , [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ])  (A.343)

2. The null space N(A) is an (N-R)-D subspace of ℝ^{4} spanned by the N − R independent homogeneous solutions:

N(A) = span ([ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] , [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ])  (A.344)

Note that the basis vectors of N(A) are indeed orthogonal to those of R(A).

3. The column space of A is the same as the row space of A^{T}, which is the R-D subspace of R^{3} spanned by the two pivot rows of the rref of A^{T}.

C(A) = R(A^{T}) = span ([ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ] , [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ])  (A.345)

4. The left null space N(A^{T}) is a (M-R)-D subspace of ℝ^{3} spanned by the homogeneous solutions (here one 3 − 2 = 1 solution):

N(A^{T}) = span ([ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]) = c [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]  (A.346)

Again note that the basis vectors of N(A^{T}) are orthogonal to those of C(A).

In general, here are the ways to find the bases of the four subspaces:

• The basis vectors of R(A) are the pivot rows of the rref of A.

• The basis vectors of C(A) are the pivot rows of the rref of A^{T}.

• The basis vectors of N(A) are the independent homogeneous solutions of Ax = 0. To find them, reduce A to the rref, identify all free variables x_{f} corresponding to non-pivot columns, set one of them to 1 and the rest to 0, solve homogeneous system Ax = 0 to find the pivot variables x_{p} to get one basis vector. Repeat the process for each of the free variables to get all basis vectors.

• The basis vectors of R(A^{T}) can be obtained by doing the same as above for A^{T}.

Note that while the basis of R(A) are the pivot rows of the rref of A, as its rows are equivalent to those of A, the pivot columns of the rref basis of C(A) are not the basis of C(A), as the columns of A have been changed by the row deduction operations and are therefore not equivalent to the columns of the resulting rref. The columns in A corresponding to the pivot columns in the rref could be used as the basis of C(A). Alternatively, the basis of C(A) can be obtained from the rref of A^{T}, as its rows are equivalent to those of A^{T}, which are the columns of A.

We further make the following observations:

• The basis vectors of each of the four subspaces are independent, the basis vectors of R(A) and N(A) are orthogonal, and dim R(A) + dim N(A) = M. Similarly, the basis vectors of C(A) and N(A^{T}) are orthogonal, and dim C(A) + dim N(A^{T}) = N. In other words, the four subspaces indeed satisfy the following orthogonal and complementary properties:

C(A) ⊥ N(A^{T}), R(A) ⊥ N(A), C(A)⊕N(A^{T}) = R^{3}, R(A)⊕N(A) = R^{4}

(A.347)

i.e., they are orthogonal complements: C(A) = N(A)^{⊥}, R(A) = N(A)^{⊥}.

• For Ax = b to be solvable, the constant vector b on the right-hand side must be in the column space, b ∈ C(A). Otherwise the equation is not solvable, even if R < M < N. Similarly, for A^{T} x = c to be solvable, c must be in the row space c ∈ R(A). In the examples above, both b and c are indeed in their corresponding column spaces:

b = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = 3 [ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ]+2 [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ] ∈ C(A), c = [ \begin{matrix}3 \\ 11 \\ 19 \\ 27 \end{matrix} ] = 3 [ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ]+11 [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ] ∈ R(A)

(A.348)

But as b = [1, 3, 5]^{T} ∉ C(A) and c = [1, 1, 2, 3]^{T} ∉ R(A), the corresponding systems have no solutions.

• All homogeneous solutions of Ax = 0 are in the null space x_{h} ∈ N(A), but in general the particular solutions x_{p} ∈ R^{N} = R(A) ⊕ N(A) are not necessarily in the row space R(A). In the example above, x_{p} ∈ R^{N} is a linear combination of the R = 2 basis vectors of R(A) and N −R = 2 basis vectors of N(A):

x_{p} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ] = (0.1 [ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ] + 0.2 [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ])+(−0.3 [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] − 0.4 [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ]) = x′_{p}+x′′_{p}

(A.349)

where

x′_{p} = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ] ∈ R(A), x′′_{p} = [ \begin{matrix}−1.1 \\ 1.8 \\ −0.3 \\ −0.4 \end{matrix} ] ∈ N(A)  (A.350)

are the projections of x_{p} onto R(A) and N(A), respectively, and x′_{p} is another particular solution without any homogeneous component that satisfies Ax = b, while x′′_{p} is a homogeneous solution satisfying Ax = 0.

• All homogeneous solutions of A^{T} y = c are in the left null space y_{h} ∈ N(A^{T}), but in general the particular solutions y_{p} ∈ C(A) ⊕ N(A^{T}) = R^{M} are not necessarily in the column space. In the example above, y_{p} ∈ R^{M} is a linear combination of the R = 2 basis vectors of C(A) and M − R = 1 basis vector of N(A^{T}):

y_{p} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ] = (2 [ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ] + 1.5 [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ])−2.5 [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ] = [ \begin{matrix}2 \\ 1.5 \\ 2.5 \end{matrix} ]−2.5 [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ] = y′_{p}+y′′_{p}

(A.351)

where y′_{p} = [2, 1.5, 2.5]^{T} ∈ C(A) is a particular solution (without any homogeneous component) that satisfies A^{T} y = c.

Here is a summary of the four subspaces associated with an M by N matrix A of rank R.

dimR(A) dimN(A) dimC(A) dimR(A^{T}) solvability of Ax = b
M = N = R  R  0  R  0  unique solution x = A^{−1}b
M > N = R  R  0  R  M − R  over-constrained, solvable if b ∈ C(A)
N > M = R  R  N − R  R  0  under-constrained, infinite solutions x = x_{p} + N(A)
R < min(M, N)  R  N − R  R  M − R  solvable if b ∈ C(A), infinite solutions

(A.352)

The figure below illustrates a specific case with M = N = 3 and R = 2. As b ∉ C(A), the system can only be approximately solved to find Ax = p ∈ C(A), which is the projection of b onto the column space C(A). The error ||e|| = ||p − b|| = ||Ax_{p} − b|| is minimized, x_{p} is the optimal approximation. We will consider ways to obtain this optimal approximation in the following sections.

A.20 Pseudo-Inverse Solutions Based on SVD

In the previous section we obtained the solution of the equation Ax = b together with the bases of the four subspaces of A based its rref. Here we will consider an alternative and better way to solve the same equation and find a set of orthogonal bases that also span the four subspaces, based on the pseudo-inverse and the singular value decomposition (SVD) of A. The solution obtained this way is optimal in some certain sense as shown below.

Consider the SVD of an M × N matrix A of rank R ≤ min(M, N):

A = UΣV^{T}  (A.353)

where

Σ = [ \begin{matrix}σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{M×N}  (A.354)

is an M × N matrix with R non-zero singular values σ_{i} = \sqrt{λ_{i}} (i = 1, …, R) of A along the diagonal (starting from the top-left corner), while all other components are zero, and U = [u_{1}, ⋯, u_{M}] and V = [v_{1}, ⋯, v_{N}] are two orthogonal matrices of dimensions M × M and N × N respectively. The column vectors u_{i} (i = 1, ⋯, M) and v_{j} (j = 1, ⋯, N), are called the left and right singular vectors of A, respectively, and they can be used as the orthonormal bases to span respectively ℝ^{M} = C(A) ⊕ N(A^{T}) and its subspaces C(A) and N(A^{T}), and ℝ^{N} = C(A) ⊕ N(A) and its subspaces C(A) and N(A).

To see this, we rewrite the SVD of A as:

A = [c_{1}, ⋯, c_{N}] = UΣV^{T}

= [u_{1}, · · · · · · · · · , u_{M}] [ \begin{matrix}σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{M×N} [ v_{1}^{T} ; ⋮ ; ⋮ ; ⋮ ; v_{N}^{T} ]

= ∑_{k=1}^{R} σ_{k} (u_{k}v_{k}^{T} ) = ∑_{k=1}^{R} σ_{k} u_{k}[v_{1k}, ⋯, v_{Nk}] = [∑_{k=1}^{R}(σ_{k}v_{1k}) u_{k}, ⋯, ∑_{k=1}^{R}(σ_{k}v_{Nk}) u_{k}] (A.355)

Each column vector of A can be expressed as a linear combination of the first R columns of U corresponding to the non-zero singular values:

c_{j} = ∑_{k=1}^{R}(σ_{k}v_{jk}) u_{k} (j = 1, ⋯, N),  (A.356)

Taking transpose on both sides of the SVD we also get:

(A^{T} ) = [r_{1}, ⋯, r_{M}] = VΣ^{T} U^{T}
= ∑_{k=1}^{R} σ_{k} (v_{k}u_{k}^{T} ) = ∑_{k=1}^{R} σ_{k} v_{k}[u_{1k}, ⋯, u_{Mk}] = [∑_{k=1}^{R}(σ_{k}u_{1k}) v_{k}, ⋯, ∑_{k=1}^{R}(σ_{k}u_{Mk}) v_{k}] (A.357)

i.e., each row vector of A can be expressed as a linear combination of the first R columns of V corresponding to the non-zero singular values:

r_{i} = ∑_{k=1}^{R}(σ_{k}u_{ik}) v_{k} (i = 1, ⋯, M)  (A.358)

We therefore see that the R columns of U and V corresponding to the non-zero singular values span respectively the column space C(A) and row space R(A):

C(A) = span(u_{1}, ⋯, u_{R}),  R(A) = span(v_{1}, ⋯, v_{R})  (A.359)

and the remaining M − R columns of U and N − R columns of V corresponding to the zero singular values span respectively N(A^{T}) orthogonal to C(A), and N(A) orthogonal to R(A):

N(A^{T}) = span(u_{R+1}, ⋯, u_{M}), N(A) = span(v_{R+1}, ⋯, v_{N}) (A.360)

Subspace  Definition  Dimension  Basis
C(A) ⊆ R^{M}  column space (image) of A  R  Rcolumns ofUcorresponding to non-zero singular values
N(A^{T}) ⊆ R^{M}  left null space of A^{T}  M − R  M − Rcolumns ofUcorresponding to zero singular values
R(A) ⊆ ℝ^{N}  row space (image) of A^{T}  R  Rcolumns ofVcorresponding to non-zero singular values
N(A) ⊆ ℝ^{N}  null space of A  N − R N − Rcolumns of V corresponding to zero singular values
R^{N} = N(A) ⊕ R(A)  domain of A  N  all N columns of V
R^{M} = N(A^{T}) ⊕ C(A)  codomain of A  M

all M columns of U

(A.361)

The SVD method can be used to find the pseudo-inverse of an M × N matrix A of rank R ≤ min(M, N):

A^{−} = VΣ^{−}U^{T}  (A.362)

where both A^{−} and

Σ^{−} = [ \begin{matrix}1/σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1/σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{N×M}  (A.363)

are N × M matrices.

We further note that matrices U and V are related to each other by:

\{ \begin{matrix}A = UΣV^{T} \\ A^{−} = VΣ^{−}U^{T} \end{matrix}  or  \{ \begin{matrix}AV = UΣ \\ A^{−}U = VΣ^{−} \end{matrix}  (A.364)

or in vector form:

\{ \begin{matrix}Av_{i} = σ_{i}u_{i} (i = 1, ⋯, R) \\ A^{−}u_{i} = v_{i}/σ_{i} (i = 1, ⋯, R) \end{matrix}  \{ \begin{matrix}Av_{i} = 0 (i = R + 1, ⋯, N) \\ A^{−}u_{i} = 0 (i = R + 1, ⋯, M) \end{matrix}

(A.365)

indicating how the individual columns are related. We see that the last N − R columns v_{R+1}, ⋯, v_{N} of V form an orthogonal basis of N(A); and the last M − R columns u_{R+1}, ⋯, u_{M} of U form an orthogonal basis of N(A^{T}).

We now show that the optimal solution of the linear system Ax = b can be obtained based on the pseudo-inverse of A:

x_{svd} = A^{−}b = VΣ^{−}U^{T} b  (A.366)

Specially, if M = N = R, then A^{−} = A^{−1}, and x_{svd} = A^{−1}b is the unique and exact solution. In general when M ≠ N or R < min(M, N), the solution may not exist, or it may not be unique, but x_{svd} is still an optimal solution in two ways, from the perspective of both the domain and codomain of the linear mapping Ax, as shown below.

• In domain ℝ^{N}:

Pre-multiplying V^{T} = V^{−1} on both sides of the pseudo-inverse solution x_{svd} = VΣ^{−}U^{T} b given above, we get:

V^{T} x_{svd} = [v_{1}, ⋯, v_{N}]^{T} x_{svd} = Σ^{−}U^{T} b = Σ^{−}[u_{1}, ⋯, u_{M}]^{T} b, (A.367)

or in component form:

[ v_{1}^{T} x_{svd} ⋮ v_{R}^{T}x_{svd} v_{R+1}^{T}x_{svd} ⋮ v_{N}^{T}x_{svd} ] = [ \begin{matrix}1/σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1/σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ] [ u_{1}^{T} b ⋮ u_{R}^{T}b u_{R+1}^{T}b ⋮ u_{M}^{T}b ] = [ u_{1}^{T} b/σ_{1} ; ⋮ ; u_{R}^{T}b/σ_{R} ; 0 ; ⋮ ; 0 ]

(A.368)

The first R components v_{i}^{T} x_{svd} = u_{i}^{T} b/σ_{i} (i = 1, ⋯, R) are the projection of x_{svd} onto R(A) spanned by {v_{1}, ⋯, v_{R}} corresponding to the non-zero singular values, while the last N − R components v_{i}^{T} x_{svd} = 0 (i = R + 1, ⋯, N) are the projection of x_{svd} onto N(A) spanned by {v_{R+1}, ⋯, v_{N}} corresponding to the zero singular values. We see that the pseudo-inverse solution x_{svd} is entirely in R(A), containing no homogeneous component in N(A). In other words, x_{svd} has the minimum norm (closest to the origin) compared to any other possible solution x_{p}, such as those found previously based on the rref of A, containing a non-zero homogeneous component x_{h} ∈ N(A):

||x_{svd}|| ≤ ||x_{p}|| = ||x_{svd} + x_{h}||  (A.369)

As N(A) ⊥ R(A), x_{svd} ∈ R(A) is the projection of any such solution x_{p} onto R(A). The complete solution can be found as

x_{c} = x_{svd} + N(A) = x_{svd} + ∑_{j=R+1}^{N} c_{j}v_{j}  (A.370)

for any set of coefficients c_{j}.

• In codomain ℝ^{M}:

We now consider the result b_{svd} = Ax_{svd} produced by the pseudo-inverse solution x_{svd}, which, as a linear combination of the columns of A, is in its column space C(A):

b_{svd} = Ax_{svd} = AA^{−}b = UΣV^{T} VΣ^{−}U^{T} b = UΣΣ^{−}U^{T} b (A.371)

Pre-multiplying U^{T} = U^{−1} on both sides we get:

U^{T} b_{svd} = ΣΣ^{−}U^{T} b  (A.372)

or in component form:

[ u_{1}^{T} b_{svd} ⋮ u_{R}^{T}b_{svd} u_{R+1}^{T}b_{svd} ⋮ u_{M}^{T}b_{svd} ] = [ \begin{matrix}1 & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1 \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ] [ u_{1}^{T} b ⋮ u_{R}^{T}b u_{R+1}^{T}b ⋮ u_{M}^{T}b ] = [ u_{1}^{T} b ; ⋮ ; u_{R}^{T}b ; 0 ; ⋮ ; 0 ]

(A.373)

The first R components u_{i}^{T} b_{svd} = u_{i}^{T} b (i = 1, ⋯, R) are the projection of b_{svd} onto C(A) spanned by {u_{1}, ⋯, u_{R}} corresponding to the non-zero singular values, while the last M − R components u_{i}^{T} b_{svd} = 0 (i = R + 1, ⋯, M) are the projection of b_{svd} onto N(A^{T}) spanned by {u_{R+1}, ⋯, u_{M}} corresponding to the zero singular values. We see that b_{svd} and b have the same projection onto C(A). If b ∈ C(A), i.e., u_{i}^{T} b = 0 for all i = R + 1, ⋯, M, then b_{svd} = Ax_{svd} = b and x_{svd} is an exact solution. But if b ∉ C(A), as it contains non-zero components in N(A^{T}), then no solution exists. However, as N(A^{T}) ⊥ C(A), b_{svd} ∈ C(A) is the projection of b onto C(A), the error ||b − b_{svd}|| is minimized when compared with that of any other approximate solution x_{p} ≠ x_{svd}:

||b − b_{svd}|| = ||b − Ax_{svd}|| ≤ ||b − Ax_{p}||  (A.374)

In this sense, x_{svd} is the optimal.

Summarizing the two aspects above, we see that the pseudo-inverse solution x_{svd} is optimal in the sense that both its norm ||x_{svd}|| and its error ||Ax_{svd} −b|| are minimized.

• If the solution x_{svd} is not unique because N(A) ≠ ∅, then the complete solution can be found by adding the entire null space to it: x_{c} = x_{svd} + N(A).

• If no solution exists because b ∉ C(A), then x_{svd} is the optimal approximate solution with minimum error.

Example: Given the same system considered in previous examples

Ax = [ \begin{matrix}1 & 2 & 3 & 4 \\ 4 & 3 & 2 & 1 \\ −2 & 1 & 4 & 7 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ]  (A.375)

we will now solve it using the pseudo-inverse method. We first find SVD of A in terms of the following matrices:

U = [u_{1}, u_{2}, u_{3}] = [ \begin{matrix}−0.535 & −0.218 & 0.817 \\ −0.267 & −0.873 & −0.408 \\ −0.802 & 0.436 & −0.408 \end{matrix} ]  (A.376)

V = [v_{1}, v_{2}, v_{3}, v_{4}] = [ \begin{matrix}0.000 & −0.837 & 0.374 & −0.400 \\ −0.267 & −0.478 & −0.797 & 0.255 \\ −0.535 & −0.120 & 0.472 & 0.691 \\ −0.802 & 0.239 & −0.049 & −0.546 \end{matrix} ] (A.377)

Σ = [ \begin{matrix}10.0 & 0 & 0 & 0 \\ 0 & 5.477 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ] ,  Σ^{−} = [ \begin{matrix}0.1 & 0 & 0 \\ 0 & 0.183 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix} ]  (A.378)

A^{−} = VΣ^{−}U^{T} = [ \begin{matrix}0.000 & −0.837 & 0.374 & −0.400 \\ −0.267 & −0.478 & −0.797 & 0.255 \\ −0.535 & −0.120 & 0.472 & 0.691 \\ −0.802 & 0.239 & −0.049 & −0.546 \end{matrix} ] [ \begin{matrix}0.100 & 0 & 0 \\ 0 & 0.183 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}−0.535 & −0.267 & 0.802 \\ −0.218 & −0.873 & 0.436 \\ 0.817 & −0.408 & −0.408 \end{matrix} ]

= [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ]

The particular solution of the system Ax = b with b = [3, 2, 4]^{T} ∈ C(A) is

x_{svd} = A^{−}b = [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ] [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]

= −0.535 [ \begin{matrix}0.000 \\ −0.267 \\ −0.535 \\ −0.802 \end{matrix} ] − 0.120 [ \begin{matrix}−0.837 \\ −0.478 \\ −0.120 \\ 0.239 \end{matrix} ] = c_{1}v_{1} + c_{2}v_{2}

which is in R(A) spanned by the first R = 2 columns v_{1} and v_{2}, perpendicular to N(A) spanned by the last N − R = 2 columns v_{3} and v_{4}. Note that this solution x_{svd} = [0.1, 0.2, 0.3, 0.4]^{T} is actually the first component x′_{p} ∈ R(A) of the particular solution x_{p} = [−1, 2, 0, 0] found in the previous section by Gauss-Jordan elimination, which is not in R(A). Adding the null space to the particular solution, we get the complete solution:

x_{c} = x_{svd}+N(A) = x_{p}+c_{1}v_{3}+c_{2}v_{4} = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]+c_{1} [ \begin{matrix}0.374 \\ −0.797 \\ 0.472 \\ −0.049 \end{matrix} ]+c_{2} [ \begin{matrix}−0.400 \\ 0.255 \\ 0.691 \\ −0.546 \end{matrix} ]

(A.379)

If the right-hand side is replaced by b = [1, 3, 5]^{T} ∉ C(A), no solution exists. However, we can still find the pseudo-inverse solution as the optimal approximate solution:

x_{svd} = A^{−}b = [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ] [ \begin{matrix}1 \\ 3 \\ 5 \end{matrix} ] = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]  (A.380)

which is the same as the solution for b = [3, 2, 4]^{T}, indicating [3, 2, 4]^{T} happens to be the projection of [1, 3, 5]^{T} onto C(A) spanned by u_{1} and u_{2}. The result produced by x_{svd} is b′ = Ax_{svd} ∈ C(A) is the projection of b onto C(A), with a minimum error distance ||e|| = ||b − b′||, indicating x_{svd} is the optimal approximate solution.

Homework 3:

1. Use Matlab function svd(A) to carry out the SVD of the coefficient matrix A of the linear system in Homework 2 problem 3,

A = [ \begin{matrix}−1 & 3 & 4 & 1 \\ 2 & −4 & 3 & 2 \\ 1 & −1 & 7 & 3 \end{matrix} ]  (A.381)

Find U, V, Σ. Verify that U and V are orthogonal, i.e., U^{T} U = I and V^{T} V = I.

2. Obtain Σ^{−}, i.e, find the transpose of Σ and then replace each singular value by its reciprocal, verify that ΣΣ^{−} = I and Σ^{−}Σ = I. Then find the pseudoinverse A^{−} = UΣ^{−}V^{T}.

3. Use Matlab function pinv(A) to find the pseudo-inverse Σ^{−} and A^{−}. Compare them to what you obtained in the previous part.

4. Identify the bases of the four subspaces R(A), N(A), C(A) = R(A^{T}), and N(A^{T}) based on U and V. Verify that these bases and those you found previously (problem 3 of Homework 2) span the same spaces, i.e., the basis vectors of one basis can be written as a linear combination of those of the other, for each of the four subspaces.

5. Use the pseudo-inverse A^{−} found above to solve the system Ax = b. Find the two particular solutions x_{p1} and x_{p2} corresponding to two different right-hand side b_{1} = [1, 2, 3]^{T} and b_{2} = [2, 3, 2]^{T}.

6. How are the two results x_{p1} and x_{p2} related to each other? Give your explanation. Why can’t you find a solution when b = [2, 3, 2]^{T} but SVD method can?

7. Verify that x_{p} ⊥ N(A). Also find the error (or residual) e = b−Ax_{p} for each of the two results above. Verify that e ⊥ C(A), if e ≠ 0.

8. In Homework 2 you used row reduction method to solve the system Ax = b_{1} and you should have found a particular solution x_{1} = [5, 2, 0, 0, ]^{T}. Also it is obvious to see that another solution is x_{2} = [0, 0, 0, 1]^{T}. Show that the projection of these solutions onto R(A) spanned by the first two columns of V is the same as the particular solution x_{p} found by the SVD method.

9. Show that b_{1} = [1, 2, 3]^{T} is the projection of b_{2} = [2, 3, 2]^{T} onto C(A), the 2-D subspace spanned by the first two columns of U. Can you also show that it is the projection of b onto C(A) spanned by the basis you obtained in Homework 2 (not necessarily orthogonal)?

10. Give an expression of the null space N(A), and then write the complete solution in form of x_{c} = x_{p}+N(A). Verify any complete solution so generated satisfies Ax_{c} = b_{1}.

A.21 Rank-Nullity Theorem

Definition: The number of independent rows in a matrix is called the row rank, the number of independent columns in a matrix is called the column rank.

Theorem: The row rank and column rank of any matrix are the same.

Proof: Let the column rank of the following m × n matrix A be r

A = [a_{1}, ⋯, a_{r}, a_{r+1}, ⋯, a_{n}] = [A_{1}, A_{0}]  (A.382)

where A_{1} = [a_{1}, ⋯, a_{r}] is an m × r sub-matrix formed by the r linearly independent columns in A, assumed to be the first r columns (without loss of generality). Now consider the linear combination of another set of r n-D vectors {A_{1}^{T} a_{1}, ⋯, A_{1}^{T} a_{r}}:

∑_{i=1}^{r} c_{i}A_{1}^{T} a_{i} = A_{1}^{T} ∑_{i=1}^{r} c_{i}a_{i} = A_{1}^{T} A_{1}c = 0  (A.383)

which is assumed to be zero. As the r × r coefficient matrix A_{1}^{T} A_{1} has full rank r (A_{1} is formed by r independent vectors), we must have c = [c_{1}, ⋯, c_{r}]^{T} = 0, indicating {A_{1}^{T} a_{1}, ⋯, A_{1}^{T} a_{r}} are independent. However, as the linear combinations of r of the rows of A, these r independent vector A_{1}^{T} a_{i} (i = 1, ⋯, r) are in the row space of A. We therefore know that the row rank of A is no smaller than its column rank.

As the same argument can also be made for A^{T}, i.e., the column rank of A is no smaller than its row rank, we conclude that the column and row ranks of A must be the same. QED

Rank-nullity theorem: The sum of the rank of an m × n matrix A and the dimension of its nullspace N(A), the nullity of A, is n:

rank(A) + dim(N(A)) = rank(A) + Nullity(A) = n  (A.384)

Proof:

Let A = [a_{1}, ⋯, a_{r}, a_{r+1}, ⋯, a_{n}] be an m × n matrix of rank r. We assume, without loss of generality, the first r columns are independent, and the last n − r columns are dependent, and write it as A = [A_{1}, A_{0}]. Here

A_{1} = [a_{1}, ⋯, a_{r}]_{m×r}  (A.385)

is formed by the r independent columns of A, i.e., if A_{1}x = 0, then x = 0; and

A_{0} = [a_{r+1}, ⋯, a_{n}]_{m×(n−r)}  (A.386)

is formed by the n − r dependent columns of A, i.e., any of them can be written as a linear combination of those independent columns a_{1}, ⋯, a_{r} in A_{1}:

a_{r+k} = ∑_{i=1}^{r} c_{ik}a_{i} = [a_{1}, ⋯, a_{r}] [ c_{1k} ; ⋮ ; c_{rk} ] = A_{1}c_{k}, (k = 1, ⋯, n − r) (A.387)

where c_{k} = [c_{1k}, ⋯, c_{rk}]^{T} is a coefficient vector. We can put all these n − r equations together to write them in matrix form as

A_{0} = [a_{r+1}, ⋯, a_{n}] = A_{1}[c_{1}, ⋯, c_{n−r}] = A_{1}C  (A.388)

where C = [c_{1}, ⋯, c_{n−r}] is an r×(n−r) matrix composed of the n−r coefficient vectors. The matrix A can now be written as

A = [A_{1}, A_{0}] = [A_{1}, A_{1}C]  (A.389)

Define an n × (n − r) matrix

X = [ \begin{matrix}−C \\ I \end{matrix} ] = [ \begin{matrix}−c_{1} & ⋯ & −c_{n−r} \\ e_{1} & ⋯ & e_{n} \end{matrix} ] = [ \begin{matrix}−c_{1} & ⋯ & −c_{n−r} \\ 1 & ⋯ & 0 \\ 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 \\ 0 & ⋯ & 1 \end{matrix} ] (A.390)

where I is an (n − r) × (n − r) identity matrix composed of n standard basis vectors e_{1}, ⋯, e_{n}. Now we have

AX = [A_{1}, A_{1}C] [ \begin{matrix}−C \\ I \end{matrix} ] = −A_{1}C + A_{1}C = O  (A.391)

indicating any column x in X is a solution of the homogeneous equation Ax = 0, i.e., x is in the nullspace N(A). Also, if we let Xu = 0, we get

Xu = [ \begin{matrix}−C \\ I \end{matrix} ] u = [ \begin{matrix}−Cu \\ u \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ] = 0  (A.392)

i.e., u = 0, indicating all columns in X are independent.

On the other hand, we can also show that any solution of the homogeneous equation Ax = 0 must be a linear combination of the columns of X. Assuming x is such a solution, i.e.,

Ax = 0 = [A_{1}, A_{1}C] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = A_{1}(x_{1} + Cx_{2})  (A.393)

As all columns in A are independent, we must have

x_{1} + Cx_{2} = 0, or x_{1} = −Cx_{2}  (A.394)

and we get

x = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−Cx_{2} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−C \\ I \end{matrix} ] x_{2} = Xx_{2}  (A.395)

indicating x is a linear combination of the n − r columns in X.

Based on the two results obtained above, we can now conclude that the nullspace N(A) is spanned by the n − r independent column vectors in X, i.e., the dimension of N(A) is n − r, and rank(A) + dim(A) = n.

A.22 Taylor Series Expansion

• Single-variable functions:

A single-variable function f(x) can be expanded around a given point x by the Taylor series:

f(x + δx) = f(x) + f′(x)δx + \frac{1}{2!}f′′(x)δx^{2} + \frac{1}{3!}f′′′(x)δx^{3} + ⋯ (A.396)

When δx is small, the higher order terms can be neglected so that the function can be approximated as a quadratic function

f(x+δx) = f(x)+f′(x)δx+\frac{1}{2!}f′′(x)δx^{2}+ε(δx^{3}) ≈ f(x)+f′(x)δx+\frac{1}{2!}f′′(x)δx^{2}

(A.397)

or even a linear function

f(x + δx) = f(x) + f′(x)δx + ε(δx^{2}) ≈ f(x) + f′(x)δx  (A.398)

• Multi-variable scalar-valued functions:

A multi-variable function f(x_{1}, ⋯, x_{n}) = f(x) can also be expanded by the Taylor series:

f(x+δx) = f(x)+∑_{j=1}^{n} \frac{∂f(x)}{∂x_{j}}δx_{j}+\frac{1}{2} ∑_{i=1}^{n} ∑_{j=1}^{n} \frac{∂^{2}f(x)}{∂x_{i}∂x_{j}} δx_{i}δx_{j}+\frac{1}{3!} ∑_{i=1}^{n} ∑_{j=1}^{n} ∑_{k=1}^{n} \frac{∂^{3}f(x)}{∂x_{i}∂x_{j}∂x_{k}} δx_{i}δx_{j}δx_{k}+⋯

(A.399)

which can be expressed in vector form as:

f(x + δx) = f(x) + g_{f}^{T} δx + \frac{1}{2}δx^{T} H_{f} δx + ⋯  (A.400)

where δx = [δx_{1}, ⋯, δx_{n}]^{T}, while g_{f} and H_{f} are the gradient vector and the Hessian matrix for the first and second order derivatives of the function, respectively:

g_{f}(x) = ▽f(x) = \frac{d}{dx}f(x) = [ ⋮_{\frac{∂fx)}{∂x_{n}}}^{\frac{∂fx)}{∂x_{1}}} ]_{n×1} ,  (A.401)

H_{f}(x) = \frac{d}{dx}g_{f}(x) = \frac{d^{2}}{dx^{2}} f(x) = [ \begin{matrix}\frac{∂^{2}f(x)}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x)}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{n}^{2}} \end{matrix} ]_{n×n} = H_{f}^{T}

(A.402)

Note that H_{f}^{T} is symmetric as ∂^{2}f(x)/∂x_{i}∂x_{j} = ∂^{2}f(x)/∂x_{j}∂x_{i}.

When δx is small (i.e., ||δx|| is small), then f(x+δx) can be approximated by a quadratic function with a third order error term

f(x + δx) = f(x) + g_{f}^{T} δx + \frac{1}{2}δx^{T} H_{f} δx + ε(||δx||^{3})  (A.403)

or even a linear function with a second order error term

f(x + δx) = f(x) + g_{f}^{T} δx + ε(||δx||^{2})  (A.404)

• Multi-variable vector-valued functions:

A set of m multi-variable functions f_{1}(x), ⋯, f_{m}(x) can be expressed as a vector function

f(x) = [f_{1}(x), ⋯, f_{m}(x)]^{T}  (A.405)

The Taylor expansion of the ith component is:

f_{i}(x + δx) = f_{i}(x) + g_{i}^{T} δx + \frac{1}{2}δx^{T} H_{i} δx + ε(||δx||^{3}) (i = 1, ⋯, m)

(A.406)

The first two terms of these m components can be written in vector form:

f(x+δx) ≈ [ f_{1}(x) ; ⋮ ; f_{m}(x) ]+[ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} & ⋯ & \frac{∂f_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{m}}{∂x_{1}} & ⋯ & \frac{∂f_{m}}{∂x_{n}} \end{matrix} ] [ δx_{1} ; ⋮ ; δx_{n} ] = f(x)+J_{f} (x) δx

(A.407)

where J_{f} (x) is the Jacobian matrix defined over the vector function f(x):

J_{f} (x) = \frac{d}{dx}f(x) = [\frac{∂f}{∂x_{1}} , ⋯, \frac{∂f}{∂x_{n}} ] = [ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} & ⋯ & \frac{∂f_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{m}}{∂x_{1}} & ⋯ & \frac{∂f_{m}}{∂x_{n}} \end{matrix} ]_{m×n} (A.408)

Note that the Jacobian matrix is not necessarily square as m and n may not be the same. The 2nd and higher order terms can no longer expressed in matrix.

Note that the Hessian matrix of a function f(x) can be obtained as the Jacobian matrix of the gradient vector of f(x):

g_{f}(x) = [ ⋮_{\frac{∂f}{∂x_{n}}}^{\frac{∂f}{∂x_{1}}} ] , J_{g}(x) = \frac{d}{dx}g_{f}(x) = [ \begin{matrix}\frac{∂^{2}f}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f}{∂x_{n}^{2}} \end{matrix} ] = H_{f} (x)

(A.409)

A.23 Normal Direction of a Plane

In a d-dimensional space,

• A point is represented as a column vector x = [x_{1}, ⋯, x_{d}]^{T} containing its d coordinates;

• A surface (or supersurface if d > 3) is represented by an equation f(x_{1}, ⋯, x_{d}) = f(x) = 0. Alternatively, by solving the equation above for one of the variables, we get x_{d} = g(x_{1}, ⋯, x_{d−1}), representing the height along the direction of x_{d} of a surface defined over a d − 1 dimensional space.

• A plane (or superplane if d > 3) is a special surface described by a linear equation:

f(x) = w^{T} x + b = ∑_{i=1}^{d} w_{i}x_{i} + b = 0  (A.410)

Without loss of generality, we can assume w is normalized with length ||w|| = \sqrt{w^{T} w} = 1 (as otherwise we can divide the equation above by ||w||).

• The d-dimensional space is divided by the plane into two regions:

R_{+} = {x | w^{T} x + b > 0}, and R_{−} = {x | w^{T} x + b < 0} (A.411)

• The noamal direction of a plane f(x) = w^{T} x + b = 0 is w, i.e., the projection of any point x on the plane onto normal direction w is constant:

p_{w}(x) = w^{T} x = −b  (A.412)

• Solving the linear equation w^{x} + b = 0 for any of the d variables, e.g., x_{d}, we get an alternative representation for the plane, as a function defined over the d-1 dimensional space spanned by the remaining d − 1 variables {x_{1}, ⋯, x_{d−1}}:

x_{d} = g(x_{1}, ⋯, x_{d−1}) = −\frac{1}{w_{d}} (∑_{i=1}^{d−1} w_{i}x_{i} + b)  (A.413)

In particular, when x_{1} = ⋯ = x_{d−1} = 0, we get x_{d} = −b/w_{d}, the intercept of the plane on the axis of x_{d}.

• When the plane is represented in the form of x_{d} = g(x_{1}, ⋯, x_{d−1}) = −(∑_{i=1}^{d−1} w_{i}x_{i}+ b)/w_{d}, the normal direction is represented the following, a scaled version of w:

[−\frac{w_{1}}{w_{d}} , ⋯, −\frac{w_{d−1}}{w_{d}} , −1]^{T} = −\frac{1}{w_{d}} [w_{1}, ⋯, w_{d}]^{T} = −\frac{1}{w_{d}} w  (A.414)

Example: The following equation defines a plane in 3-D space spanned by x, y and z:

f(x, y, z) = ax + by + cz + d = x + 2y + 3z + 4 = 0

where a = 1, b = 2, c = 3 and d = 4. Solving for z we get

z = g(x, y) = −\frac{1}{c}(ax + by + d) = −\frac{1}{3}(x + 2y + 4)

When x = y = 0, we get z = −c/d = −4/3, the intercept of the plane on z-axis. The normal direction of the plane is composed of the coefficients of the three variable x, y and z in f(x, y, z):

n = [a, b, c]^{T} = [1, 2, 3]^{T}  (A.415)

or the coefficients of the two variables x and y of g(x, y), with the last component −1:

n = [−\frac{a}{c}, −\frac{b}{c}, −1]^{T} = −\frac{1}{c}[a, b, c]^{T} = −\frac{1}{3}[1, 2, 3]  (A.416)

A.24 Convexity

We first consider some concepts regarding convexity:

• Convex set or convex region: a region in an N-D space in which every point on the straight line between any pair of two points in the region is also inside the region.

• Convex hull or convex envelope, convex closure of a shape: the smallest convex set containing the shape.

• Convex function: a real-valued function f(x) with the property that its epigraph or supergraph, the region above its graph, is a convex set.

• Concave function: if f(x) is convex, then −f(x) is concave.

A multivariate function f(x) is convex if and only if

f(λx_{1} + (1 − λ)x_{2}) ≤ λf(x_{1}) + (1 − λ)f(x_{2})  (A.417)

Equivalently, f(x) is concave if and only if

f(λx_{1} + (1 − λ)x_{2}) ≥ λf(x_{1}) + (1 − λ)f(x_{2})  (A.418)

The convex function is strict if equality in the relaship never holds. Geometrically, the line segment connecting two points (x_{1}, f(x_{1})) and (x_{2}, f(x_{2})) is above the graph of the convex function f(x).

Convexity plays an important role in optimization problems, due to its nice property that if a function f(x) is differentiable and strictly convex, then any point x^{∗} with a zero gradient g_{f}(x^{∗}) = 0 is the unique global minimum of the function. Minimizing a convex objective function, or equivalently maximizing a concave objective function, over a convex domain is called convex optimization.

Here are some examples and properties of convex functions:

• Single-variable functions: x^{2}, |x|, e^{ax}, − log(x)

• Any norm of a function or vector is convex. The intersection of the epigraphs is convex.

• The intersection of two convex sets is convex. Also,

• Affine function: f(x) = w^{T} x + b is both convex and concave but not strict, as both Eq. (A.417) and Eq. (A.418) become equality when applied to the function.

• If function f(x) is convex, then a f(x) + b is also convex, if a ≥ 0 so that the convexity is preserved.

• If two functions f(x) and g(x) are both convex, then their linear combination is also convex if the coefficients are no negative:

h(x) = af(x) + bg(x), a, b ≥ 0  (A.419)

This can be proven by applying the definition directly to h(x).

• A linear combination of n convex function is also convex if all coefficients are nonnegative:

f(x = ∑_{i=1}^{n} w_{i}f_{i}(x), w_{1}, ⋯, w_{n} ≥ 0  (A.420)

• Quadratic functions: f(x) = x^{T} Qx + w^{T} x + b is – convex if Q ≥ 0 is positive semidefinite;

– strictly convex if Q > 0 is positive definite;

– concave if Q ≤ 0 is negative semidefinite;

– strictly concave if Q < 0 is negative definite;

If the Hessian matrix H_{f}(x), the second order derivative, of f(x) exists and is positive semidefinite, i.e., H_{f}(x) ≥ 0 for its entire domain, then f(x) is convex. As simple examples in 1-D space, f_{1}(x) = x^{2} and f_{2}(x) = e^{x} are both convex, and their Hessian, the second order derivatives, f′′_{1}(x) = 2 and f′′_{2}(x) = e^{x} are both greater than zero.

Specifically, let x_{1} and x_{2} be any two points inside the domain of f(x). Then f(x) is convex if for any c ∈ [0, 1] the following holds:

f(cx_{1} + (1 − c)x_{2}) ≤ c f(x_{1}) + (1 − c)f(x_{2})  (A.421)

where cx_{1} + (1 − c)x_{2} is any point along the line segment between x_{1} and x_{2}, at which the function value is less than the linear interpolation of the two function values f(x_{1}) and f(x_{2}). If the equality does not hold, then f(x) is strictly convex. We can also define a concave function as the negative of a convex function. If f(x) is convex, then g(x) = −f(x) is concave, for which the following inequality holds:

g(cx_{1} + (1 − c)x_{2}) ≥ c g(x_{1}) + (1 − c)g(x_{2})  (A.422)

A convex function f(x) has the property that any local minimum is also a global minimum. Moreover, if f(x) is strictly convex, then it has a single global minimum.

A.25 Jensen Inequality

For a convex function g(x), the following holds:

g (∑_{i=1}^{n} λ_{i}x_{i}) ≤ ∑_{i=1}^{n} λ_{i}f(x_{i})  (A.423)

For a concave function g(x), the following holds:

g (∑_{i=1}^{n} λ_{i}x_{i}) ≥ ∑_{i=1}^{n} λ_{i}g(x_{i})  (A.424)

Proof:

Let λ_{1} and λ_{2} be two arbitrary nonnegative real numbers satisfying λ_{1}+λ_{2} = 1. Then by the definition of convexity, we have

f(λ_{1}x_{1} + λ_{2}x_{2}) ≤ λ_{1}f(x_{1}) + λ_{2}f(x_{2})  (A.425)

This results of two points can be generalized to n points by induction. We assume the inequality holds for k points:

f (∑_{i=1}^{k} λ_{i}x_{i}) ≤ ∑_{i=1}^{k} λ_{i}f(x_{i})  (A.426)

Now consider the case of k+1 terms with coifficients satisfying λ_{1}+⋯+λ_{k+1} = 1:

f (∑_{i=1}^{k+1} λ_{i}x_{i}) = f (λ_{1}x_{1} + (1 − λ_{1}) ∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i})

≤ λ_{1}f(x_{1}) + (1 − λ_{1})f (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i})  (A.427)

But as

∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} = 1  (A.428)

we have the following due to the assumption above:

f (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i}) ≤ ∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} f(x_{i})  (A.429)

Substituting this back we get the desired inequality:

f (∑_{i=1}^{k+1} λ_{i}x_{i}) ≤ λ_{1}f(x_{1}) + (1 − λ_{1}) (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} f(x_{i})) = ∑_{i=1}^{k+1} λ_{i}f(x_{i}) (A.430)

If x is a random vector that takes any of the n values x_{i} with probability P_{i} satisfying P_{1} + ⋯ + P_{n} = 1, then we have

f(E(x)) = f (∑_{i=1}^{k} x_{i}P_{i}) ≤ ∑_{i=1}^{k} f(x_{i})P_{i} = E(f(x))  (A.431)

This result can be generalized to the case where x is a random vector taking continuous values with E[x] = ∫ x p(x)dx.

A.26 Sensitivity and Conditioning

An important concern in numerical analysis is how sensitive the result produced by an algorithm is with respect to the noise in the input data (e.g., observational error) and computational errors (e.g., truncation error). To address this issue, we model a generic algorithm treated as a function with both input and outputp most generally defined (e.g., real or complex, scalar, vector, matrix, function, etc.), and describe its sensitivity to noise as below:

• A function is well conditioned (well behaved) if a small change in the input causes a small change in the output.

• A function is ill conditioned (ill behaved) if a small change in the input causes a large change in the output.

How well or ill a function is conditioned can be measured quantitatively by the absolute or relative condition numbers defined below:

• The absolute condition number is the upper bound of the ratio between the change in output and change in input:

κ̂ ≥ \frac{||change in output||}{||change in input||}  (A.432)

• The relative condition number is the upper bound of the ratio between the relative change in output and relative change in input:

κ ≥ \frac{||change in output||/||output||}{||change in input||/||input||}  (A.433)

where ||x|| denotes the norm of x. As the relative condition number compares the normalized changes in both input and output, its value is invariant to the specific units used to measure them. Obviously the smaller the condition number, the better the function is conditioned.

Consider specifically the condition numbers of some different systems.

System with Scalor input and output:

We first consider evaluating a scalor function y = f(x) given x. The condition number is:

κ̂ = lim_{ϵ→0} sup_{|δx|≤ϵ} \frac{|δf(x)|}{|δx|},  κ = lim_{ϵ→0} sup_{|δx|≤ϵ} \frac{|δf(x)|/|f(x)|}{|δx|/|x|},  (A.434)

Here sup is the symbol for supremum (least upper bound) and |x| is either the absolute value of x if it is real or the modulus of x if it is complex. Note that in general the condition number is a function of x.

When δx is small, f(x + δx) can be approximated by the first two terms of its Taylor series:

f(x + δx) = f(x) + f′(x)δx + f′′(x)\frac{δx^{2}}{2} + ⋯ ≈ f(x) + f′(x)δx (A.435)

then

δy = f(x + δx) − f(x) ≈ f′(x)δx  (A.436)

Taking the absolute value or modulus on both sides, we get

|δy| ≈ |f′(x)δx| ≤ |f′(x)| |δx|  (A.437)

and the absolute condition number is:

κ̂ ≈ |δy|/|δx| ≈ |f′(x)|  (A.438)

Dividing both sides by |y| = |f(x)|, we furhter get

\frac{|δy|}{|y|} ≈ \frac{|f′(x)| |δx|}{|f(x)|} = \frac{|x| |f′(x)|}{|f(x)|} \frac{|δx|}{|x|}  (A.439)

and the relative condition number:

κ = \frac{|δy|/|y|}{|δx|/|x|} ≈ \frac{|x| |f′(x)|}{|f(x)|} = \frac{|f′(x)|}{|f(x)|/|x|}  (A.440)

We next consider solving the equation f(x) = 0 to find x treated as the output that satisfies y = f(x) = 0 treated as the input. The problem can be converted into the evaluation of the function x = g(y) = f^{−1}(y) at y = 0. The absolute condition number of this function g(y) = f^{−1}(y) is:

κ̂ = |g′(y)| = |\frac{d}{dy} [f^{−1}(y)]| = \frac{1}{|f′(x)|}  (A.441)

which is the reciprocal of the absolute condition number |f′(x)| of the previous evaluating problem, evaluated at the root x^{∗} satisfying f(x^{∗}) = 0. We see that the larger |f′(x)|, or the smaller g′(y) = 1/|f′(x)|, the better conditioned the problem is.

In the figure below shows two functions |f′_{1}(x)| > |f′_{2}(x)| for all x. For evaluating function y = f(x), f_{1}(x) is more ill-conditioned than f_{2}(x), but for solving f(x) = 0, f_{1}(x) is more well-conditioned than f_{2}(x).

Consider as an example the following two problems both associated with a linear function y = f(x) = ax + b:

• Evaluation of y given x:

The condition number is κ = δy/δx. If x is changed to x + δx due to noise, then y becomes ax + b + aδx = y + δy, where δy = aδx, i.e., κ = a. The larger a, the more ill-conditioned the system is.

• Solving y = f(x) = 0 for x given y = 0:

The condition number is κ = δx/δy. The given equation can be solved for x:

x = g(y) = \frac{1}{a}(y − b)  (A.442)

If y is changed to y+δy due to noise, then x becomes (y−b)/a+δy/a = x+ δx, where δx = δy/a, i.e., κ = 1/a. The smaller a, the more ill-conditioned the system is.

Systems with vector input and output

The discussions above can be generalized to a function y = f(x) with vector input x = [x_{1}, ⋯, x_{N}]^{T} and vector output y = [y_{1}, ⋯, y_{M}]^{T}. Again consider approximating the function by the first two terms in the Taylor series:

f(x + δx) ≈ f(x) + J_{f}(x)δx  (A.443)

where J_{f}(x) is the Jacobian matrix of the function. We get

||δy|| = ||f(x + δx) − f(x)|| ≈ ||J_{f}(x)δx|| ≤ ||J_{f}(x)|| ||δx||  (A.444)

and the absolute condition number:

κ̂ = \frac{||δy||}{||δx||} ≈ ||J_{f}(x)||  (A.445)

Dividing both sides of the equation above by ||y|| = ||f(x)|| we further get

\frac{||δy||}{||y||} ≈ \frac{||J_{f}(x)δx||}{||f(x)||} ≤ \frac{||x|| ||J_{f}(x)||}{||f(x)||} \frac{||δx||}{||x||}  (A.446)

and we can find the relative condition number as

κ = \frac{||δy||/||y||}{||δx||/||x||} ≈ \frac{||x|| ||J_{f}(x)||}{||f(x)||}  (A.447)

Consider as an example the following two problems both associated with a linear function:

y = f(x) = Ax  (A.448)

• Evaluating y = Ax given x:

The Taylor series of this linear function is

f(x + δx) = A(x + δx) = Ax + J_{f} δx = Ax + δAx  (A.449)

i.e., δy = Aδx and ||δy|| ≤ ||A|| ||δx||. We get the upper bound of the absolute condition number:

κ̂(A) = \frac{||δy||}{||δx||} ≤ ||A||  (A.450)

The relative condition number of this linear system is

κ(A) = \frac{||x|| ||J_{f}(x)||}{||f(x)||} = ||A||\frac{||x||}{||Ax||}  (A.451)

We also have

||x|| = ||A^{−1}Ax|| ≤ ||A^{−1}|| ||Ax||, i.e., ||Ax|| ≥ \frac{||x||}{||A^{−1}||} (A.452)

Substituting this into the expression of κ(A) above we get its upper bound:

κ(A) = ||A||\frac{||x||}{||Ax||} ≤ ||A|| ||A^{−1}||\frac{||x||}{||x||} = ||A|| ||A^{−1}||  (A.453)

Note that κ is no less than 1:

κ(A) = ||A^{−1}|| ||A|| ≥ ||A^{−1}A|| = ||I|| = 1  (A.454)

• Solving Ax = y for x as output given y as input:

We assume A is a full-rank square matrix and therefore invertible, so that there exists a unique solution: x = g(y) = A^{−1}y. The absolute condition number is κ̂(A) = ||J_{g}(x)|| = ||A^{−1}||.

To find the relative condition number κ(A), consider δx in the output x caused by δA as well as δy:

(A + δA)(x + δx) = Ax + Aδx + δAx + δAδx = y + δy  (A.455)

Subtracting Ax = y from both sides we get

Aδx + δAx + δAδx = δy  (A.456)

i.e.,

δx = A^{−1}(δy − δAx − δAδx)  (A.457)

Taking norm on both sides, we get

||δx|| ≤ ||A^{−1}|| ||δy − (δAx + δAδx)||
≤ ||A^{−1}|| (||δy|| + ||(δAx + δAδx)||)
≤ ||A^{−1}|| (||δy|| + ||δA|| ||x|| + ||δA|| ||δx||) (A.458)

where we have used ||a − b|| ≤ ||a|| + ||b||, ||a + b|| ≤ ||a|| + ||b||, and ||Ax|| ≤ ||A|| ||x||.

Subtracting ||A^{−1}|| ||δA|| ||δx|| from both sides

(1 − ||A^{−1}|| ||δA||) ||δx|| ≤ ||A^{−1}|| (||δy|| + ||δA|| ||x||)  (A.459)

and dividing both sides by (1 − ||A^{−1}|| ||δA||) ||y|| (assuming not zero), we get

\frac{||δx||}{||x||} ≤ \frac{||A^{−1}||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||x||} + ||δA||) = \frac{||A^{−1}|| ||A||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||x|| ||A||} + \frac{||δA||}{||A||})
≤ \frac{||A^{−1}|| ||A||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||y||} + \frac{||δA||}{||A||}) = \frac{κ(A)}{1 − κ(A)||δA||/||A||} (\frac{||δy||}{||y||} + \frac{||δA||}{||A||}) (A.460)

where we have used ||y|| = ||Ay|| ≤ ||A|| ||x||, and κ(A) = ||A^{−1}|| ||A||.

In particular,

– if δA = 0,

\frac{||δx||}{||x||} ≤ κ(A)\frac{||δy||}{||y||}  (A.461)

– if δy = 0,

\frac{||δx||}{||x||} ≤ \frac{κ(A)}{1 − κ(A)||δA||/||A||} \frac{||δA||}{||A||}  (A.462)

We see that the relative condition number is κ(A) = ||A|| ||A^{−1}||, same for both problems of evaluation y = Ax and solving equation Ax = y.

We further consider the spectral norm defined as the greatest singular value of the matrix ||A|| = σ_{max}. Then ||A^{−1}|| = 1/σ_{min} is the reciprocal of the smallest singular value of A. Now the relative condition number is

κ(A) = ||A|| ||A^{−1}|| = \frac{σ_{max}}{σ_{min}}  (A.463)

We see that κA is large if its σ_{max} and σ_{min} are far apart in values, but it is small otherwise. It is therefore a measurement of how close A is to singularity. If A is singular, one or more of its singular values are zero, i.e., σ_{min} = 0, then κ(A) = ∞, and the system Ax = y is worst conditioned. On the other hand, if A = I, i.e., all singular values equal to 1, then κ(I) = 1, the smallest condition number possible, and the system Ix = y is best conditioned.

In Matlab, the function cond(A,p) generates the condition number of matrix A based on its p-norm. When p = 2, this is the ratio between the greatest and smallest singular values. The following Matlab commands give the same result:

• cond(A,2)
• norm(A,2)*norm(inv(A),2)
• s=svd(A), s(1)/s(end)

Consider whether the following examples are needed...

We now further consider the worst case scenarios for both evaluating the matrix-vector product y = Ax and solving the linear equation system Ax = y.

Consider the SVD decomposition A = UΣV^{∗} and assume the singular values in Σ are sorted in descending order: σ_{1} ≥ ⋯ ≥ σ_{n}. The SVD of the inverse is A^{−1} = VΣ^{−1}U^{∗}.

• The worst case for evaluating Ax = y:

If x = v_{n} is the minimum right singular vector corresponding to the smallest singular value σ_{n} = σ_{min}. The equation becomes:

y = Ax = (UΣV^{∗}) v_{n} = UΣ [ v_{1}^{∗} ; ⋮ ; v_{n}^{∗} ] v_{n} = U [ \begin{matrix}σ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & σ_{n} \end{matrix} ] [ 0 ⋮ 0 1 ]

= [u_{1}, ⋯, u_{n}] [ 0 ; ⋮ ; 0 ; σ_{n} ] = σ_{n}u_{n}  (A.464)

and

y + δy = A(x + δx) = σ_{n}u_{n} + Aδx  (A.465)

The absolute error ||δy|| = ||Aδx|| may be small if δx is small, but the relative error ||δy||/||y|| may be large as ||y|| = |σ_{n}| ||u|| is small when σ_{n} = σ_{min} is small (e.g., close to zero).

• The worst case for solving y = Ay:

If y = u_{1} is the maximum left singular vector corresponding to the largest singular value σ_{1} = σ_{max}. The equation becomes:

x = A^{−1}y = VΣ^{−1}U^{∗}u_{1} = VΣ^{−1} [ u_{1}^{∗} ; ⋮ ; u_{n}^{∗} ] u_{1} = V [ \begin{matrix}1/σ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 1/σ_{n} \end{matrix} ] [ 1 0 ⋮ 0 ]

= [v_{1}, ⋯, v_{n}] [ 1/σ_{1} ; 0 ; ⋮ ; 0 ] = \frac{1}{σ_{1}} v_{1}  (A.466)

and

x + δx = A^{−1}(y + δy) = \frac{1}{σ_{1}} v_{1} + A^{−1}δy  (A.467)

The absolute error ||δx|| = ||A^{−1}δx|| may be small if δy is small, but the relative error ||δx||/||x|| may be large as ||x|| = ||u||/|σ_{1}| is small when σ_{1} = σ_{max} is large.

Example 0:

A = [ \begin{matrix}5.4321 & 1.2345 \\ 10.8643 & 2.4689 \end{matrix} ] ,  (A.468)

The singular values are σ_{1} = 12.45633, σ_{2} = 0.00005

||A|| = σ_{1} = 12.45633,  ||A^{−1}|| = 1/σ_{1} = 18684.68,  κ(A) = ||A|| ||A^{−1}|| = 12.45633×18684.68 = 232742.6

(A.469)

As A is near singular, its condition number is large, indicating Ax = y is an ill-conditioned problem. It is easy to see that the problem of solving Ay = x is ill-conditioned. Assuming δA = 0, we have

y + δy = A^{−1}(x + δx), δy = A^{−1}δx  (A.470)

The change in output y is

||δy|| ≤ ||A^{−1}|| ||δx|| = 18684.68 ||δx||  (A.471)

which is 18684.68 times greater than any change δx in the input. However, for an evaluation problem to find y = Ax given x,

y + δy = A(x + δx), δy = Aδx  (A.472)

The change in output y is

||δy ≤ ||A|| ||δx|| = 12.45633 ||δx||  (A.473)

which is not too much greater than the change δx in the input.

Example 1: Consider solving the linear system Ax = b with

A = \frac{1}{2} [ \begin{matrix}3.0 & 2.0 \\ 1.0 & 4.0 \end{matrix} ] , A^{−1} = [ \begin{matrix}0.8 & −0.4 \\ −0.2 & 0.6 \end{matrix} ]  (A.474)

The singular values of A are σ_{1} = 2.5583 and σ_{2} = 0.9772, the condition number is

κ = ||A|| ||A^{−1}|| = \frac{σ_{1}}{σ_{2}} = 2.618  (A.475)

which is small, indicating this is a well-behaved system. Given two similar inputs b_{1} = [1, 1]^{T} and b_{2} = [0.99, 1.01]^{T} with δb = [0.01, −0.01]^{T}, we find the corresponding solutions:

x_{1} = A^{−1}b_{1} = [ 0.4//0.4 ] , x_{2} = A^{−1}b_{2} = [ \begin{matrix}0.388 \\ 0.408 \end{matrix} ] , δx = [ \begin{matrix}0.012 \\ −0.008 \end{matrix} ]

(A.476)

We have

||δb|| = 0.0141, ||b_{1}|| = 1.4142, ||δx|| = 0.0144, ||x_{1}|| = 0.5657 (A.477)

and

\frac{||δx||/||x_{1}||}{||δb||/||b_{1}||} = \frac{0.0255}{0.01} = 2.5495  (A.478)

Example 2:

A = \frac{1}{2} [ \begin{matrix}1.000 & 1.000 \\ 1.001 & 0.999 \end{matrix} ] , A^{−1} = [ \begin{matrix}−999 & 1000 \\ 1001 & −1000 \end{matrix} ]  (A.479)

The singular values of A are σ_{1} = 1.0 and σ_{2} = 0.0005, the condition number is

κ = ||A|| ||A^{−1}|| = \frac{σ_{1}}{σ_{2}} = 2000  (A.480)

indicating matrix A is close to singularity, and the system is ill-conditioned. The solutions corresponding to the same two inputs b_{1} = [1, 1]^{T} and b_{2} = [0.99, 1.01]^{T} are

x_{1} = A^{−1}b_{1} = [ \begin{matrix}1 \\ 1 \end{matrix} ] , x_{2} = A^{−1}b_{2} = [ \begin{matrix}20.99 \\ −19.01 \end{matrix} ] , δx = [ \begin{matrix}−19.99 \\ 20.01 \end{matrix} ]

(A.481)

We can further find

||δb|| = 0.0141, ||b_{1}|| = 1.4142, ||δx|| = 28.2843, ||x_{1}|| = 1.4142 (A.482)

and the condition number is:

\frac{||δx||/||x_{1}||}{||δb||/||b_{1}||} = \frac{200}{0.01} = 2000  (A.483)

We see that a small relative change ||δb||/||b_{1}|| = 0.01 in the input caused a huge change ||δx||/||x_{1}|| = 20 in the output (2000 times greater).

Example 3: Consider

A = [ \begin{matrix}3.1 & 2.0 \\ 6.0 & 3.9 \end{matrix} ] = UΣV^{∗} = [ \begin{matrix}−0.458 & −0.889 \\ −0.889 & 0.458 \end{matrix} ] [ \begin{matrix}8.051 & 0.0 \\ 0.0 & 0.011 \end{matrix} ] [ \begin{matrix}−0.839 & −0.544 \\ −0.544 & 0.839 \end{matrix} ]

(A.484)

with

||A|| = σ_{1} = 8.05, ||A^{−1}|| = 1/σ_{2} = 89.46, κ = ||A|| ||A^{−1}|| = 720.22

(A.485)

As A is near singular, its condition number κ is large, indicating that the problem associated with this A is ill-conditioned. We will now specifically consider both problems of evaluating and solving the linear system.

• Solve Ay = x for y given x:

y + δy = A^{−1}(x + δx), δy = A^{−1}δx  (A.486)

The absolute condition number is κ̂ = ||A^{−1}|| = 89.46. The change in output y is

||δy|| ≤ ||A^{−1}|| ||δx|| = κ̂ ||δx|| = 89.46 ||δx||  (A.487)

which is κ̂ = 89.46 times greater than any change δx in the input. We further consider the worst case when x = u_{max} and the relative error:

x = u_{1} = [ \begin{matrix}−0.458 \\ −0.889 \end{matrix} ] , y = A^{−1}x = [ \begin{matrix}−0.104 \\ −0.068 \end{matrix} ]  (A.488)

Even for a small change δx in the input, causing a small change δy in the output:

δx = [ \begin{matrix}0.01 \\ −0.01 \end{matrix} ] , δy = A^{−1}δx = [ \begin{matrix}0.656 \\ −1.011 \end{matrix} ]  (A.489)

the ratios of the absolute and relative errors are:

\frac{||δy||}{||δx||} = \frac{1.205}{0.014} ≈ 85.21,  \frac{||δy||/||y||}{||δx||/||x||} = \frac{9.702}{0.014} ≈ 686  (A.490)

• Evaluate y = Ax given x:

y + δy = A(x + δx), δy = Aδx  (A.491)

The absolute condition number is κ̂ = ||A|| = 8.05. The absolute change in output y is

||δy|| ≤ ||A|| ||δx|| = 8.051 ||δx||  (A.492)

which is not too much greater than the absolute change δx in the input. However, a much greater relative error will result if we consider the worst case when x = v_{min}:

x = v_{2} = [ \begin{matrix}−0.5444 \\ 0.8388 \end{matrix} ] , y = Ax = [ \begin{matrix}−0.0099 \\ 0.0051 \end{matrix} ]  (A.493)

Even for a small change δx in the input, causing a small change δy in the output:

δx = [ \begin{matrix}0.01 \\ −0.01 \end{matrix} ] , δy = Aδx = [ \begin{matrix}0.011 \\ 0.021 \end{matrix} ]  (A.494)

The ratios of the absolute and relative errors are:

\frac{||δy||}{||δx||} ≈ 1.68, \frac{||δy||/||y||}{||δx||/||x||} = \frac{2.1207}{0.0141} ≈ 150  (A.495)

Although the ratio of the absolute errors is small, the ratio of the relative errors is large.

A.27 Some Useful Inequalities

• The interpolation inequality for e^{x}

e^{ta+(1−t)b} ≤ te^{a} + (1 − t)e^{b}, t ∈ [0, 1], a, b ≥ 0  (A.496)

Proof: Evaluating the exponential function y = f(x) = e^{x} at two points x = a and x = b we get two points in the x-y plane (a, e^{a}) and (b, e^{b}). The parametric expression of the secant line through these two points is

(ta + (1 − t)b, te^{a} + (1 − t)e^{b}), (0 ≤ t ≤ 1)  (A.497)

The parametric expression of the function y = e^{x} between these two points is:

(ta + (1 − t)b, e^{ta+(1−t)b}), (0 ≤ t ≤ 1)  (A.498)

As the y = e^{x} is convex downward (or concave upward), it is never higher than the secant line function in the interval a ≤ x ≤ b, and we have

e^{ta+(1−t)b} ≤ te^{a} + (1 − t)e^{b}  (A.499)

The equality holds only when t = 0 or t = 1.

• Young’s inequality For any two real positive values p, q > 0 satisfying

\frac{1}{p} + \frac{1}{q} = 1, q = \frac{p}{p − 1}, p = \frac{q}{q − 1}  (A.500)

and any two real numbers α and β, we have

αβ ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q}  (A.501)

Proof: Substituting a = p log α and b = q log β into the interpolation inequality for f(x) = e^{x} above, we get:

e^{tp log α+(1−t)q log β} = e^{tp log α}e^{(1−t)q log β} = α^{tp}β^{(1−t)q}
≤ te^{p log α} + (1 − t)e^{q log β} = tα^{p} + (1 − t)β^{q}

We further let t = 1/p, and get 1 − t = 1 − 1/p = 1/q, the above becomes Young’s inequality:

αβ ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q}  (A.502)

• Holder’s inequality

| ∑_{i} x_{i}y_{i}| ≤ ∑_{i} |x_{i}y_{i}| ≤ ||x||_{p}||y||_{q}  (A.503)

Proof: Applying Young’s inequality to the following

α = \frac{|x_{i}|}{||x||_{p}} = \frac{|x_{i}|}{(∑_{i} |x_{i}|^{p})^{1/p}} , β = \frac{|y_{i}|}{||y||_{q}} = \frac{|y_{i}|}{(∑_{i} |y_{i}|^{q})^{1/q}}  (A.504)

we get

αβ = \frac{|x_{i}| |y_{i}|}{||x||_{p} ||y||_{q}} ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q} = \frac{1}{p} \frac{|x_{i}|^{p}}{∑_{i} |x_{i}|^{p}} + \frac{1}{q} \frac{|y_{i}|^{q}}{∑_{i} |y_{i}|^{q}}  (A.505)

Taking summation on both sides we get

\frac{∑_{i} |x_{i}| |y_{i}|}{||x||_{p} ||y||_{q}} ≤ \frac{1}{p} + \frac{1}{q} = 1  (A.506)

i.e.,

∑_{i} |x_{i}y_{i}| ≤ ∑_{i} |x_{i}| |y_{i}| ≤ ||x||_{p} ||y||_{q}  (A.507)

In particular when p = 2, q = p/(p − 1) = 2, we have

| < x, y > | = | ∑_{i} x_{i}\overline{y}_{i}| = | ∑_{i} x_{i}y_{i}| ≤ ∑_{i} |x_{i}y_{i}| ≤ ||x||_{2} ||y||_{2} (A.508)

This is the Cauchy-Schwarz inequality.

• Minkowski’s inequality

||x + y||_{p} ≤ ||x||_{p} + ||y||_{p}, p ≥ 1  (A.509)

Proof:

||x + y||_{p}^{p} = ∑_{i} |x_{i} + y_{i}|^{p} = ∑_{i} |x_{i} + y_{i}| |x_{i} + y_{i}|^{p−1} ≤ ∑_{i}(|x_{i}| + |y_{i}|) |x_{i} + y_{i}|^{p−1}
= ∑_{i} |x_{i}| |x_{i} + y_{i}|^{p−1} + ∑_{i} |y_{i}| |x_{i} + y_{i}|^{p−1}

Applying Holder’s inequality to each of the two terms on the right-hand side, we get

||x + y||_{p}^{p} ≤ ||x||_{p} (∑_{i} |x_{i} + y_{i}|^{(p−1)q})^{1/q} + ||y||_{p} (∑_{i} |x_{i} + y_{i}|^{(p−1)q})^{1/q}

(A.510)

But as 1/p + 1/q = 1, and

q = p/(p − 1), 1/q = (p − 1)/p, (p − 1)q = p  (A.511)

the above becomes:

||x+y||_{p}^{p} ≤ ||x||_{p} (∑_{i} |x_{i} + y_{i}|^{p})^{(p−1)/p}+||y||_{p} (∑_{i} |x_{i} + y_{i}|^{p})^{(p−1)/p} = (||x||_{p} + ||y||_{p}) ||x+y||_{p}^{p−1}

(A.512)

Multiplying both sides by ||x + y||_{p}^{1−p}, we get Minkowski’s inequality.

Notes

Chapter 2

1 Lewis Fry Richardson (1881–1953).

Author index

Peterson, K., 42 Tranah, D.A., 42 Young, P.D.F., 42

Subject index

diffraction, 42 force

hydrodynamic, 42 interactive, 42