Introduction to Machine Learning

From Math to Code

Ruye Wang

Contents

Preface page viii Notation xv

Part I Mathematical Fundations 1

1  Solving Equations  5

1.1 Linear Equation Systems 6 1.2 The Bisection and Secant Methods 8

1.2.1 The Bisection Search 8 1.2.2 The Secant Method 10

1.3 Fixed-point Iteration 11 1.4 Newton-Raphson Method (Univariate) 22 1.5 Newton-Raphson Method (Multivariate) 26 Problem 33 1.A Order of Convergence of the Secant Method 34 1.B Order of Convergence of the Newton-Raphson Method 35 1.C Proofs of the Fixed Point and Contraction Mapping Theorems 36

2  Unconstrained Optimization  38

2.1 Optimization by Solving Equations 39 2.2 Newton’s Method 40 2.3 Gradient Descent Method 46 2.4 Line minimization 49 2.5 Quasi-Newton Methods 56 2.6 Conjugate gradient method 63 2.7 Issues of Local/Global Minimum 76 Problems 77

3  Constrained Optimization  79

3.1 Optimization with Equality Constraints 80 3.2 Optimization with Inequality Constraints 83 3.3 Duality and KKT Conditions 88 3.4 Linear Programming (LP) 92 3.5 The Simplex Algorithm 98 3.6 Quadratic Programming (QP) 104 3.7 Interior Point Methods 106 Problems 114

Part II Regression 117

4  Bias-Variance Tradeoff and Overfitting vs Underfitting  120

4.1 Supervised Learning as Optimization 120 4.2 Bias-Variance Tradeoff 124 4.3 Cross-Validation 128 4.4 Regularization and Ensemble Learning 129

5  Linear Regression  131

5.1 Linear Least Squares (LLS) Regression 131 5.2 Ridge Regression 142 5.3 Regression Based on Basis Functions 145 5.4 Bayesian Regression 153 Problems 161

6  Nonlinear Regression  164

6.1 Nonlinear Least Squares Regression 164 6.2 Parameter Estimation by Natural Gradient Descent 171 Problems 172

7  Logistic and Softmax Regression  173

7.1 Logistic Regression and Binary Classification 173 7.2 Softmax Regression for Multiclass Classification 181 Problems 189

8  Gaussian Process Regression and Classification  191

8.1 Gaussian Process Regression 191 8.2 Gaussian Process Classifier – Binary 197 8.3 Gaussian Process Classifier – Multi-Class 204 Problems 213

Part III Feature Extraction 215

9  Feature Selection  218

9.1 Distances and Separability Measurements 218 Problems 222

10  Principal Component Analysis  225

10.1 Covariance and Correlation 225 10.2 Karhunen-Loève Transformation 228 10.3 Optimality of KLT 230 10.4 Geometric Interpretation of the KLT 233 10.5 Computation of the KLT 235 10.6 Comparison with Other Orthogonal Transforms 237 10.7 Application to Image Data 239 10.8 PCA for Feature Extraction 245 Problems 249

11  Variations of PCA  252

11.1 Kernel Methods 252 11.2 Kernel PCA 256 11.3 Factor Analysis and Expectation Maximization 259 11.4 Probabilistic PCA 266 11.5 Classical Multidimensional Scaling 271 11.6 t-Distributed Stochastic Neighbor Embedding 276 Problems 283

12  Independent Component Analysis  285

12.1 Independence and Non-Gaussianity 285 12.2 Preprocessing and Whitening 287 12.3 Non-Gaussianity and FastICA 288 12.4 Likelihood and Independence Maximization 292 Problems 298

Part IV Classification 299

13  Statistic Classification  303

13.1 Discriminative vs. Generative Methods for Classification 303 13.2 K Nearest Neighbors and Minimum Distance Classifiers 305 13.3 Naive Bayes Classification 307 13.4 Adaptive Boosting 319 Problems 326

14  Support Vector machine  332

14.1 Maximum Margin and Support Vectors 332 14.2 Kernel Mapping 341 14.3 Soft Margin SVM 342 14.4 Sequential Minimal Optimization Algorithm 345 14.5 Multiclass Classification 354 14.6 Kernelized Bayes classifier 360 Problems 367

15  Clustering Analysis  369

15.1 K-means clustering 369 15.2 Gaussian mixture model 375 15.3 Bernoulli Mixture Model 385 Problems 390

16  Hierarchical Classifiers  392

16.1 Bottom-Up vs Top-Down Methods 392 16.2 Binary Hierarchical Classification 394 16.3 Binary Hierarchical Clustering 399 Problems 405

Part V Neural Networks 407

17  Biologically Inspired Networks  409

17.1 Biological Inspiration 410 17.2 Hebbian Learning 414 17.3 Hopfield Network 417

18  Perceptron-Based Networks  421

18.1 Perceptron 421 18.2 Back Propagation 429 18.3 Autoencoder 439 18.4 Deep Learning 448 Problems 453

19  Competition-Based Networks  456

19.1 Competitive Learning Network 456 19.2 Self-Organizing Map (SOM) 461 Problems 469

Part VI Reinforcement Learning 473

20  Introduction to Reinforcement Learning  477

20.1 Markov Decision Process 477 20.1.1 Markov Chain 477 20.1.2 Markov Reward Process 478 20.1.3 Markov Decision Process 481

20.2 Model-Based Planning 484 20.3 Model-Free Evaluation and Control 489

20.3.1 Monte Carlo (MC) Algorithms 494 20.3.2 Temporal Difference (TD) Algorithms 497 20.3.3 TD(λ) Algorithm 504

20.4 Value Function Approximation 509 20.5 Control Based on Function Approximation 514 20.6 Deep Q-learning 516 20.7 Policy Gradient Methods 520 Problems 527

Appendix A A Review of Linear Algebra 529

A.1 Inner Product Space 529 A.1.1 Vector Space 529 A.1.2 Orthogonal Basis and Gram-Schmidt Process 536

A.2 Matrices 538 A.2.1 Rank, Trace, Determinant, Transpose, and Inverse 538 A.2.2 Normal, Unitary, and Similar Matrices 540 A.2.3 Positive/Negative (Semi)-Definite Matrices 541 A.2.4 Woodbury Matrix Identity and Sherman-Morrison Formula 542 A.2.5 Inverse and Determinant of Partitioned Symmetric Matrices 543 A.2.6 Unitary Transform 545

A.3 Eigenvalues and Eigenvectors 547 A.3.1 Eigenvalue Decomposition 547 A.3.2 Generalized Eigenvalue Problem 551 A.3.3 Rayleigh Quotient 553 A.3.4 Normal Matrices and Diagonalizability 553 A.3.5 Singular Value Decomposition 555

A.4 Vector and Matrix Calculations 558 A.4.1 Vector Norms 558 A.4.2 Matrix Norms 562 A.4.3 Vector and Matrix Differentiation 570

A.5 The Fundamental Theorem of Linear Algebra 571 A.5.1 Rank-Nullity Theorem 571 A.5.2 Solving Linear Equation Systems 573

A.6 Pseudo-Inverse 585 A.6.1 Pseudo-Inverse 585 A.6.2 Pseudo-Inverse Solutions Based on SVD 587

A.7 Taylor Series Expansion 594 A.8 Miscellaneous 596

A.8.1 Centering Matrix 596 A.8.2 Normal Direction of a Plane 597 A.8.3 Convexity 599 A.8.4 Jensen Inequality 600 A.8.5 Sensitivity and Conditioning 601 A.8.6 Some Useful Inequalities 610

Appendix B A Review of Probability and Statistics 613

Notes 657

Preface

Machine learning (ML), as a branch of artificial intelligence (AI), has reached its maturity, due to the advancements in both computing power and new theoretical and algorithmic development in the recent decades. The technology has found many new applications in a wide spectrum of areas in both industry and our daily life, thereby generating a significant amount passion among college students as well as practicing professionals with different backgrounds to gain the knowledge and master the skill in ML, as a fascinating intersection of applied mathematics, computer science, and engineering. This book is motivated by such passion among my students during my teaching of the subject over several years, and it grew out of the lecture notes developed for the course. The book is written for upper-level undergraduate or first year graduate students in computer science, engineering, or any other relevant majors, and it can also be used as a reference for practicing professionals of different background but interested in ML.

A different Pedagogical Phhilosophy

The idea of converting notes into a textbook came from the realization that my approach of teaching the subject is not quite the same as many of the existing textbooks on the subject, which can be roughly categorized into either theoretical or practical type. Books of the theoretical type treat ML as a subject of applied mathematics, emphasizing the mathematical theory as the background of the various ML methods. Such books typically provide detailed mathematical background and rigorous derivations of the algorithms, but they pay little attention to the actual implementation of the algorithms discussed. Readers of these books may be left unsatisfied if they also want to know how the algorithms are actually implemented in code and applied to real-world problems. On the other hand, books of the practical type treat the subject of ML as a computational toolbox, which is black or a grey at best, in the sense that they emphasize mostly the application of the tools in the toolbox while paying little attention to the mathematical background of the algorithms. Such books typically rely on certain off-the-shelf commercial software packages, and they discuss mostly how the functions in some built-in library are used, but very little why the underneath algorithms work (or not work under certain conditions). Some of such books may by in the style of manual books or cooking recipes, providing little more than a set of steps to follow. While such books may serve the purpose of certain readers, others may feel unsatisfied knowing only how to use a black box blindly, when they also desire to open the box to learn its inner workings.

Different from either of these two approaches, this book is intended as neither a theoretical mathematics book nor a practical manual book, but a combination of both. It is written based on the belief that machine learning can be best learned as a subject of both applied mathematics for the theoretical background of the algorithms, and a branch of engineering for the implementation and application of the algorithms. As suggested by the subtitle From Math to Code, this book attempts to bridge the gap between the theoretical foundation of the ML methods and their algorithmic and code level implementation, so that readers seriously interested in the subject can learn to understand both why and how the ML algorithms work. Once they gain such insights of an algorithm, as well as its strength and weakness, conditions and limitations, they will be able to not only use the algorithm properly and effectively, but also possibly modify and improve it while solving their specific real world problems.

Pedagogical Feature and Code

In addition to the rigorous mathematical presentation needed for the in-depth understanding of the algorithms, also provided in the book are a large number of simple and thoroughly worked out examples following most of the algorithms discussed in the text to show how they are applied as well as implemented, and to illustrate their results, such as different effects due to different parameter settings. All such examples are based on home-brewed software in both Matlab and Python without using any existing software or libraries. Moreover, very often some essential segments of the Matlab implementation of an algorithm are included in the text following the mathematical background of the algorithm, so that the reader can see how a mathematical idea is converted from equations to code to be actually realized. It is the author’s experience and belief that studying the code implementation of an algorithm is an effective way to learn how specifically the algorithm works. By studying the code as well as the mathematics, the reader is exposed to both languages and herefore gain the valuable experience of translating mathematical ideas into code implementations. The Matlab code is written in such a way that it matches as closely as possible to the equations in the text for readability, while leaving the computational efficiency to Python, the currently dominant language for large-scale data analysis. The Python programs associated with the chapters are provided separately at: https://github.com/njkrichardson/mlam

Regarding the Mathematics

It is the author’s firm belief that to thoroughly understand the ML algorithms covered in the book, it is necessary to also learn their mathematical backgrounds, mostly in the areas of multivariable calculus, linear algebra, probability and statistics. In addition, some basic understanding of optimization (both constrained and unconstrained) and dynamic programming is needed, as many ML algorithms are essentially an optimization problem. On the other hand, the mathematical background of all potential readers of the book may span span a wide spectrum. Some readers may not necessarily have all such background knowledge and they may likely feel difficult following the mathematics in certain parts of the book. Therefore the concern regarding the proper depth and amount of mathematics in the book needs to be addressed to satisfy most of the readers, so that the mathematics in the book serves as a useful tool that helps the readers to better understand the algorithms, instead of roadblocks that discourage some readers from studying the subject of ML.

The approach taken by this book is somewhat different from many of the ML textbooks intended to seriously cover the theoretical background of the various ML algorithms, which assume the readers are either already knowledgeable in these subjects from their prior studies, or they are willing to find proper references and learn the materials on their own. Differently, this book is developed as a self-contained textbook, so that readers unfamiliar with some of the mathematical topics can find all necessary background information as well as the main ML topics in the same volumn without the need for referencing any other books. They can find a brief summary of linear algebra and probability/statistics in Appendices A and B respectively, and also a more in-depth discussion for optimization in Part I of the main text. However, such readers do not have to study all these topics as preparatory knowledge before they start to study the ML algorithms, instead, they are encouraged to refer to these mathematical reviews only when they run into difficulties while following the derivation of specific ML algorithms due to their unfamiliarity of the background mathematics. In such a case they can take a brief digression to read a few relevant paragraphs in Part I, Appendix A or B to quickly gain the necessary background knowledge and then come back right away to continue their study of the main topic. Such a brief digression within the book would be much more convenient than looking for a proper reference book and then studying a relevant section in it.

It is possible that some readers may still feel difficult understanding the mathematics in certain parts of the book due to their lack of either the relevant knowledge or enough experience to follow mathematical derivations. Such readers are encouraged to skip such parts in the text when studying the topics the first time. The study of ML, as well as any other subject for this matter, is necessarily a repetitive process which is by no means linear. A more knowledgeable and experienced reader may want to thoroughly understand an algorithm by going through all background mathematics, but this is not necessarily the only way to learn the subject. If less experienced readers run into difficulty following the derivation (occationally lengthy and tedious) of a certain algorithm when studying it the first time, they are encouraged to skip the intermediate steps and directly jump to the final result, so that they can quickly grasp the basic idea and concept of interest, while keeping the option of coming back to study the detailed mathematics later when needed.

End of Chapter Problems – Unique Approach

Different from the typical homework problems in many ML textbooks, the homework problems in this book are all about implementation of the algorithms covered in the chapter. The students are required to develop their own code (in Matlab, Python, or any other language of choice) to implement the algorithms and test them based on some given datasets. Such coding exercises are based on the belief that the ultimate goal for studying ML is to be able to apply various algorithms to solve real problems, and the ultimate test whether the students have truly understood an algorithm is to see if they are able to implement the algorithm by their own code based on the theoretical background of the algorithm, without relying on existing software readily available in some commercial libraries or packages. Such libraries and packages can be highly valuable for anyone to use for solving their practical problems, but they may not be the right tool if used as a blackbox for anyone trying to learn and understand the innerworkings of ML algorithms.

As the guiding philosophy of the book, such a belief is also put into practice during the development of the book. Most of the algorithms mentioned in the text were implemented by the author’s own Matlab code developed from scratch, and tested on some sample datasets as shown in the large number of examples in the text. Moreover, some code segments and functions flecting the essential parts of each of the algorithms also provided in the text to serve two purposes. First, according to the author’s own experience, seeing how an algorithm is actually implemented in code makes it more efficient to thorough learn the algorithm. Second, the provided code segments make the required coding homework less challenging, so that the students do not have to spend too much time to develop their code from scratch. However, a proper tradeoff needs to be made in terms of how much of code to provide. It is hoped that the amount of code in the text achieves a reasonable balance between the two extremes of providing complete code and no code at all.

Most of the homework problems ask the students to develop their own Matlab code to implement the algorithms covered in the chapters, although any other language can be used if a student so prefers. Matlab is recommended as it allows the students to concentrate on the conceptual and mathematical understanding of the algorithms without the worry of other issues such as computational efficiency and accuracy when learnig the subject the first time. Through this process the students should gain not only the in-depth understanding of the theoretical background of the algorithms in terms of why and how they work, but also first-hand experiences of the implementation of these algorithms in terms of what type of problems an algorithm is designed to solve (or unable to solve), the pros and cons of a specific algorithm in comparion to other similar algorithms, and how to choose the proper values for the parameters of the algorithm for it to achieve the best performance. Inexperienced students’ code may not be professionally developed with high computational efficiency and accuracy and good coding style, but once they have gained the insights of the algorithms, they can always take the full advantage of the professionally developed commercial software packages and apply them properly to solve real problems more effectively.

Moreover, they may potentially also be able to develop their own algorithms in the future to best solve specific problems they may have at hand.

For the students to test their own code implementing various algorithms discussed in the text, they need to apply their code to some datasets, such as those used for the examples in the book. Some of the datasets can actually be generated by the code provided in the homework, while a few others are provided on a CUP website for the book at.... The size and scale of such datasets may be too small to be practical, but they serve the purpose of debugging the code while avoiding long excution time if the code is applied to a large scale dataset. Once a piece of code is thoroughly debugged and tested, it can be applied to more realistic datasets of much larger scale, such as many of those publically available on the Internet for testing and benchmarking ML algorithms. The readers are highly encouraged to do so.

Structure and Organization of the Book

The book is organized in six parts as listed below, although not all of them need to or can be covered in a typical one-semester course. Depending on the students’ background, the chapters in Part I can be covered in the first few lectures dedicated to the background mathematics for learning methods the later chapters if the students are not familar with the topics, or, alternatively, these topics may only need to be quickly reviewed whenever they are needed while discussing the variou ML algorithms if the students are knowledgeable enough in these subjects. Part II for regression needs to be taught early on before other topics, not onlu because the mathematics is relatively more straight forward than that in the later chapters, but also some basic issues discussed in this part (such as bias-variance tradeoff and regularization in Sections 4.2 and 4.3) are relevant in general to most learning methods covered in later chapters. Part III for feature extraction can be treated as the preprocessing stage for the main learning algorithms to be discussed in Parts IV and V. Finally, as the methods and their background mathematics in Part VI for reinforcement learning are not very closely related to the materials in all previous parts, Part VI can be treated as an independent unit to be covered in the course only if time allows. Otherwise this part can be further expanded into a full course on reinforcement learning to be taught separately.

• Part I Mathematical Foundation:

This part is mostly dedicated to optimization, covering constrained as well as unconstrained methods, both of essential importance in ML, as many ML algorithms are formulated as an optimization problem to either minimize the difference between the output of a ML model and some observed training data, or to maximize the likelihood of the model parameters given the training data. Most ML textbooks do not discuss how such optimization methods are actually carried out, assuming the readers are already familiar with the subject or will pick up such knowledge on their own. However, optimization is a big subject of its own, a reader unfamiliar with it may not even know where to start. For this reason, the necessary contents in optimization relevant to ML algorithms are covered in Part I of the book for the convenience of the reader.

Also included in Part I is the topic of solving equation systems, which is one of the most fundamental computational operations for many algorithms, and it is also most closely related to optimization, in the simple sense that a variable x that maximizes function f(x) can be found by solving the equation f′(x) = 0, by, for example, the most widely used Newton’s method.

• Part II Regression Methods:

This part covers the most important regression algorithms widely used in ML, including learning and nonlinear regression, logistic and softmax regression, and Gaussian process regression. All such regression methods are closely related to classification, the core problem in ML, in the most straightforward sense that by simply thresholding the regression function, its domain is divided into two regions corresponding to two classes, i.e., the regression method can be readily used as a binary classifier. Moreover, a more sophisticated classifier can be obtained if the regression function is converted into the probability for a data sample to belong to one of the classes. This way we can not only classify a data sample into a class but also get the confidence or certainty of doing so.

• Part III Feature Selection:

This part is about feature selection and dimensionality reduction, which can be considered as the preprocessing stage of the main ML operations of classification or clustering. The core method of this part is principal component analysis (PCA), by which the dimensionality of the dataset can be reduced significantly while the information pertaining to classification or clustering contained in the data is mostly conserved. We also consider a set of variations of the PCA methods, such as kernel PCA and probabilistic PCA. In this part we also consider the method of independent component analysis (ICA), by which the independent signal components can be restored from the mixture of their linear combinations.

• Part IV Classification:

This part covers a set of core classification methods widely used in ML, including various supervised methods based on the training dataset containing a set of data samples each labeled by its class identity, such as the K-nearest neighbors (KNN) method and the minimum distance classifier, the adaptive boosting (AdaBoost), the naive Bayesian classifier, the support vector machine (SVM) with kernel mapping, and Gaussian process classifier, both binary and multiclass, and the decision tree method. In this part we also discuss unsupervised clustering methods without any training data, such as K-means method, and methods based on Gaussian and Bernoulli mixture models. All such classification and clustering can be considered as the partitioning of the multidimensional feature space into multiple regions each corresponding to one of the classes or clusters.

• Part V Neural Networks

In Part V, we introduce a set of important algorithms based on artificial neural networks inspired by the biological neural network in the brain, including the Hebbian learning for association between two sets of patterns, the Hopfield network for self-association, the perceptron network based on kernel mapping for both binary and multiclass classification, and the most popular back propagation method for multiclass classification. In this part we also briefly discuss the idea of deep learning based on multilayer back propagation learning, typified by the convolutional neural network (CNN) for image object recognition. In addition, we also discuss unsupervised neural network algorithms such as competitive learning and the closely related self-organizing maps (SOM).

We node that although the neural network methods covered in this part may seem conceptually very different from those regression and classification covered in previous parts, they may all look similar in the sense that the mathematical model of the learning taking place in a single layer of neural network is actually the same as a linear regression followed by a nonlinear mapping function, called activation function in the context of neural networks, such as a logistic mapping considered in Part II.

• Part VI Reinforcement Learning:

This lart part of the book provides an introduction to the reinforcement learning (RL), an big area of active and ongoing study that has attracted a lot of interests in both research and applications. RL is considered as the third type of learning methods fundamentally diffenent from the two traditional of learning methods, the supervised learning (regression and classification) and unsupervised learning (clustering). Instead of assigning data points in the feature space into different classes or clusters as in supervised or unsupervised learning, RL aims at optimizing a sequence of actions for an agent to take for the purpose of receiving maximum returns from a given environment, based on its interaction with the environment. In this sense, RL can also be considered as a different type of optimization achieved by unsupervised interactive learning (without labeled training data). When combined with deep learning, RL has achieved amazing results in the recent years, most famously, the software AlphaGo that beat the best human player of the board game Go.

Notation

AFM atomic force microscope
AKPZ anisotropic KPZ equation
a_{0}  lattice constant
c_{q}(ℓ) q-th order correlation function
d_{E}  embedding dimension
d_{f}  fractal dimension
L  system size
≡  defined to be equal
∼  asymptotically equal (in scaling sense)
≈  approximately equal (in numberical value)

Part I Mathematical Fundations

Many machine learning problems can be formulated as an optimization problem to build a model of a set of observed data, called training dataset or simply training set, so that the model output matches the data in an optimal manner in certain sense. Typically this is done by either minimizing or maximizing an objective function of a set of model parameters, which measures the goodness of the model. For example, in either regression or classification, the goal of a certain type of learning algorithms is to minimize the error of the model, the difference between the model prediction and the given observed data, measured in a certain way, such as the squared error. In some other type of algorithms, the goal is to maximize the likelihood of the model parameters, defined as the probability of observing the given data conditioned on these parameters.

In the typical supervised learning paradigm, the modeling process can be formulated as to construct a function y = f(x, θ) of some d variables as the components of a vector x = [x_{1}, ⋯, x_{d}]^{T}, parameterized by a set of M parameters as the components of a vector θ = [θ_{1}, ⋯, θ_{M}]^{T}. The goal is for this function to best fit a set of N data points in the training set, denoted by D = {(x_{n}, y_{n}), n = 1, ⋯, N}. While the form of the model function is usually assumed to be known based on some prior knowledge, such as a polynomial (e.g., linear or quadratic) function, or certain probability density distribution (pdf) of the data, the values of the M parameters in θ are unknown and to be estimated based on the training set D. The estimation can be carried out by various methods such as the following two that are widely used in machine learning:

• Least squares estimation (LSE):

The sum of squared error (SSE) between the model output and the desired output according to the training set is to be minimized:

J_{sse}(θ) = ∑_{n=1}^{N}[y_{n} − f((x_{n}, θ))]^{2}  (0.1)

• Maximum likelihood estimate (MLE):

The likelihood of the parameter θ, the conditional probability of observing the data in D given a specific parameter θ, is to be maximized:

J_{mle}(θ) = L(θ|D) ∝ p(D|θ) = ∏_{n=1}^{N} p(y_{n}|x_{n}, θ)  (0.2)

Solving either the minimization or maximization problem we get the optimal model parameter θ^{∗} for the model to best fit the observed data.

To prepare for the discussions of various machine learning algorithms in the future parts, such as how to find specifically the optimal θ^{∗} that minimizes/maximizes function J(θ) above, we will first consider in this part a set of typical methods for the optimization problem in the most generic form, find the optimal solution x^{∗} = [x_{1}^{2}, ⋯, x_{N}^{∗}]^{T} that either maximize or minimize a real valued multivariable function y = f(x) = f(x_{1}, ⋯, x_{N}). As maximizing f(x) is equivalent to minimizing −f(x), we typically only need to consider the minimization problem to find the optimal solution x^{∗} that minimizes f(x), denoted by

x^{∗} = arg min_{x} f(x), i.e., f(x^{∗}) = min_{x} f(x) ≤ f(x)  (0.3)

Here the function f(x) can be either linear or nonlinear, and an optimization problem is either unconstrained if the independent variable x of the function is allowed to take any value in the function domain ℝ^{N}, or constrained if x must be inside a subset of ℝ^{N}, called the feasible region. In machine learning, both linear and nonlinear methods, either constrained or not, are widely used. For example, the linear least squares method (LLS) for regression (to minimize the squared error such as that given in Eq. (0.1), Section 5.1) is solved as a linear unconstrained optimization problem, while the method of support vector machine (SVM) for classification (to maximize the margin between two classes, Section 14.1) is solved as a nonlinear constrained optimization problem.

In the following, we will review some necessary mathematical topics as the foundation for many of the learning methods to be discussed in future parts, including the numerical methods for solving equations in Chapter 1, which are not only important in their own right, but also most closely related to both unconstrained and constrained optimization problems in Chapters 2 and 3, which in turn will be needed for many important machine learning algorithms to be discussed in the future chapters.

1 Solving Equations

In this chapter, we consider some basic methods for solving a equation system in the most general form of M simultaneous equations of N variables:

\{ \begin{matrix}f_{1}(x_{1}, ⋯, x_{N}) = f_{1}(x) = 0 \\ ⋯ & ⋯ & ⋯ \\ f_{M}(x_{1}, ⋯, x_{N})) = f_{M}(x) = 0 \end{matrix}  (1.1)

Here x = [x_{1}, ⋯, x_{N}]^{T} ∈ ℝ^{N} is an N-D column vector representing a point in the N-D vector space ℝ^{N} spanned by {x_{1}, ⋯, x_{N}}, and yf_{m}(x) (m = 1, ⋯, M) is a scalar valued multivariate function that maps any point in its domain x ∈ ℝ^{N} to a scalar value y. This equation system can be represented more concisely in vector form

f(x) = 0  (1.2)

where f = [f_{1}, ⋯, f_{M}]^{T} is a column vector containing the M functions as components and 0 = [0, ⋯, 0]^{T} is a vector containing M zeros. Solving this equation system means to find a solution x^{∗} ∈ ℝ^{N} that satisfies all M equations. If such a solution does not exist, we would still like to find an approximate solution by which certain error, e.g., the sum of squared error (SSE) ||f(x)||^{2}, is minimized.

Geometrically, a function y = f(x) defined over domain ℝ^{N} is a hypersurface in an N+1 dimensional space, of which the N+1st dimension represents the function value y = f(x) corresponding to a point x ∈ ℝ^{N}. Specially the hypersurface is a curve if N = 1 or a surface if N = 2. The roots or zeros of function f(x), i.e., the solutions of equation f(x) = 0, are composed of all points on the the intersection, if exists, of the hypersurface y = f(x) and the hyperplane y = 0.

For example, in the special case of M = N = 2, functions f_{1}(x_{1}, x_{2}) and f_{2}(x_{1}, x_{2}) are two surfaces defined over the 2-D space spanned by x_{1} and x_{2}, and the roots of each of the two functions are on a curve as the intersection of the corresponding surface and the 2-D space. The solutions of the system composed of the two equations f_{1} = 0 and f_{2} = 0 are the intersection of these two curves in the 2-D space if they do intersect, otherwise no solution exists.

Example 1.1 Consider a simultaneous equation system with M = N = 2:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} − 2 = 0 \\ f_{2}(x_{1}, x_{2}) = x_{1} + x_{2} − C = 0 \end{matrix}

The first function f_{1}(x_{1}, x_{2}) is a parabolic cone in the N + 1 = 3 dimensional space centrally symmetric to the vertical axis, and its roots form a circle x_{1}^{2}+x_{2}^{2} = 2 on the 2-D plane spanned by x_{1} and x_{2} centered at the origin (0, 0) with radius 1; the second function f_{2}(x_{1}, x_{2}) is a plane in the 3-D space through the origin, and its roots form a straight line x_{2} = C − x_{1} on the 2-D plane. The solutions of the equation system are where the two curves intersect:

• If C = 0, there are two solutions (1, −1) and (−1, 1);

• If C = 2, there is only one solution (1, 1);

• If C = 3, the two curves do not intersect, i.e., no solution exists.

1.1 Linear Equation Systems

Consider a linear equation system of M equations and N variables:

\{ \begin{matrix}f_{1}(x) & = f_{1}(x_{1}, ⋯, x_{N}) & = ∑_{j=1}^{N} a_{1j}x_{j} − b_{1} = 0 \\ ⋯ & ⋯ & ⋯ \\ f_{M}(x) & = f_{1}(x_{1}, ⋯, x_{N}) & = ∑_{j=1}^{N} a_{Mj}x_{j} − b_{M} = 0 \end{matrix}  (1.3)

which can be expressed in vector form:

f(x) = Ax − b = 0  (1.4)

where

A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ] ,  x = [ x_{1} ; ⋮ ; x_{N} ] ,  b = [ b_{1} ; ⋮ ; b_{M} ]  (1.5)

In general, the existence and uniqueness of the solution can be determined based on the fundamental theorem of linear algebra (Table 3 in Section A.5), depending on the rank R of the coefficient matrix A, and whether the system is underdetermined (underconstrained) if M < N, or overdetermined (overconstrained) if M > N.

If A is a full rank square matrix with R = M = N, then its inverse A^{−1} exists and the system has a unique solution x = A^{−1}b. But if M > N = R and b is not in the column space of A, the system is overconstrained with no solutions. In this case we can still find an optimal approximate solution that minimizes the sum-of-squares error (SSE), called objective function in general optimization problems, defined as

ε(x) = \frac{1}{2}||r||^{2} = \frac{1}{2}r^{T} r = \frac{1}{2}(b − Ax)^{T} (b − Ax)
= \frac{1}{2} (b^{T} b − x^{T} A^{T} b − b^{T} Ax + x^{T} A^{T} Ax)  (1.6)

Figure 1.1 The Pseudo Inverse (Section 1.1)

where r = b − Ax is the residual of the system, and ||r||^{2} = ∑_{m=1}^{M} r_{m}^{2} (thereby the term sum of squares error). The optimal solution x^{∗} that minimizes ε(x) can be found by setting its derivative with respect to x to zero (see Section A.4.3 for differentiation with respect to a vector variable):

\frac{d}{dx}ε(x) = \frac{1}{2} \frac{d}{dx}||r||^{2} = \frac{1}{2} \frac{d}{dx} (b^{T} b − x^{T} A^{T} b − b^{T} Ax + x^{T} A^{T} Ax)
= −A^{T} b + A^{T} Ax = 0  (1.7)

Solving this matrix equation we get the optimal approximate solution

x^{∗} = (A^{T} A)^{−1} A^{T} b = A^{−}b  (1.8)

As illustrated in Fig. 1.1. Here A^{−} = (A^{T} A)^{−1} A^{T} is the pseudo-inverse (see Section A.6.1) of the non-square matrix A. Here we assume the M equations are independent, i.e., the N × N square matrix A^{T} A has a full rank R = N, i.e., it is nonsingular and invertible with all eigenvalues being non-zero.

If the M equations are barely independent, i.e., some eigenvalues of A^{T} A are very close to zero, then it is near-singular but still invertible, and some eigenvalues of its inverse (A^{T} A)^{−1} may take huge values. Consequently the result in Eq. (1.8) may vary greatly due to some small random fluctuation in either A or b. In this case, the system is said to be ill-conditioned or ill-posed, as it is prone to noise, i.e., its solution is highly sensitive to noise and therefore unreliable and unstable.

Such an ill-posed problem can be addressed by regularization, so that the solution x is forced to avoid taking huge values and therefore less sensitive to noise. This can be done by including in the objective function an additional penalty term for large values of x scaled by a parameter λ:

J(x) = \frac{1}{2}||r||^{2} + \frac{λ}{2}||x||^{2}  (1.9)

By minimizing J(x), we can obtain a solution x of small norm ||x|| as well as a small error ε(x). Just as before, the solution x can be obtained by setting the derivative of J(x) to zero

\frac{d}{dx}J(x) = −A^{T} b + A^{T} Ax + λx = 0  (1.10)

and solving the resulting equation to get:

x = (A^{T} A + λI)^{−1}A^{T} b  (1.11)

We see that even if A^{T} A is near singular, matrix A^{T} A + λI is not due to the additional term λI.

Here λ is called the hyperparameter, by adjusting it we can make a proper tradeoff between accuracy and stability:

• Small λ: the solution is more accurate but less stable as it is more prone to noise, i.e., the variance error may be large. This is called overfitting;

• Large λ: the solution is less accurate but more stable as it is less affected by noise. This is called underfitting.

The issue of overfitting versus underfitting is of essential importance in machine learning in general, it will appear repeatedly and be specifically addressed while discussing various regression and classification algorithms in the later chapters.

1.2 The Bisection and Secant Methods

Most of equation systems of interest are nonlinear. Unlike the linear equation systems which can be solved under the guidance of the fundamental theorem of linear algebra, for nonlinear equation systems, there exists neither a theory regarding the existence and uniqueness of their solutions, nor closed-form solutions in general. We therefore have to rely on numerical methods to find one or more solutions, if they do exist, in an iterative manner from certain initial guess of the solution, if the iteration converges, i.e., the difference between two consecutive results eventually approach to zero.

We first consider some methods that find a solution of a single-variable nonlinear equation f(x) = 0, by searching iteratively through a neighborhood of the domain, in which a solution is known to be located.

1.2.1 The Bisection Search

This method requires two initial guesses x_{0} < x_{1} satisfying f(x_{0})f(x_{1}) < 0. As f(x_{0}) and f(x_{1}) are on opposite sides of the x-axis y = 0, the solution x^{∗} at which f(x^{∗}) = 0 must reside somewhere in between of these two guesses, i.e., x_{0} < x^{∗} < x_{1}.

Given such two end points x_{0} and x_{1}, we could find any point x_{2} in between and use it to replace one of the end points at which the function value f(x) has the same sign as that of f(x_{2}). The new search interval is a = x_{2} − x_{0} if

Figure 1.2 Bisection Search

f(x_{2})f(x_{1}) > 0, or b = x_{1} − x_{2} if f(x_{0})f(x_{2}) > 0, either of which is smaller than the previous interval a + b = x_{1} − x_{0}. This process is then carried out iteratively until the solution is eventually approached, with a guaranteed convergence.

To avoid the worst possible case in which the solution always happens to be in the larger of the two sections a and b, we typically choose x_{2} to be the middle point x_{2} = (x_{0} + x_{1})/2 between the two end points, so that a = b = (x_{1} − x_{0})/2, i.e., the new search interval is always halved in each step of the iteration:

x_{n+1} = \frac{x_{n} + x_{n−1}}{2}  (1.12)

as illustrated in Fig. 1.2. Here, without loss of generality, we have assumed f(x_{n−1})f(x_{n+1}) > 0 and x_{n−1} is replaced by x_{n+1}.

For example, we assume the root is located at x^{∗} = 2.2 between the two initial values are x_{0} = 0 and x_{1} = 8, then we get

\begin{matrix}n & 0 & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 \\ x_{n} & 0 & 8 & 4 & 2 & 3 & 2.5 & 2.25 & 2.125 & ⋯ \\ |e_{n}| & 2.2 & 5.8 & 1.8 & 0.2 & 0.8 & 0.3 & 0.05 & 0.075 & ⋯ \end{matrix}  (1.13)

Note that the error |e_{n}| = |x_{n} − x^{∗}| does not necessarily always reduce monotonically. However, as in each iteration the search interval is always halved, the worst possible error is also halved:

|e_{n+1}| = |x_{n+1} − x^{∗}| ≤ \frac{|x_{n} − x_{n−1}|}{2} = \frac{|x_{0} − x_{1}|}{2^{n}} ≈ \frac{|e_{n}|}{2}  (1.14)

In general, to measure how quickly or slowly an iterative method converges, we consider the following limit of the ratio of errors of two consecutive iterations:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{q}} = lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|} ≤ μ  (1.15)

where q is the order of convergence and μ is the rate of convergence. The higher

Figure 1.3 The Secant Method

the order q, the more rapidly the iteration converges. For the bisection method, we have

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{q}} = lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|} ≤ μ = \frac{1}{2}  (1.16)

We see that its order of convergence is q = 1 (it converges linearly) with the rate of convergence μ = 1/2. Compared to other methods to be considered later, the linear convergence of the bisection method is rather slow, but its has the advantage that no derivative f′(x) of the given function is needed and the given function does not need to be differentiable.

1.2.2 The Secant Method

Same as in the bisection method, we assume there are two initial values x_{0} and x_{1} available, but they no longer have to satisfy f(x_{0})f(x_{1}) < 0. Here the iteration is based on the zero-crossing of the secant line passing through the two points f(x_{0}) and f(x_{1}), instead of the middle point between them. The equation for the secant is:

y = f(x_{0}) + \frac{x − x_{0}}{x_{1} − x_{0}} (f(x_{1}) − f(x_{0})) = f(x_{0}) + f̂′(x_{0})(x − x_{0})  (1.17)

where f̂′(x_{0}) is the finite difference that approximates the derivative f′(x_{0}):

f̂′(x_{0}) = \frac{f(x_{1}) − f(x_{0})}{x_{1} − x_{0}} = \frac{Δf(x_{0})}{Δx_{0}} ⟹^{Δx_{0}→0} f′(x_{0})  (1.18)

Setting y = 0 in the equation above, we get the zero crossing point:

x = x_{0} − \frac{x_{1} − x_{0}}{f(x_{1}) − f(x_{0})}f(x_{0}) = x_{0} − \frac{f(x_{0})}{f̂′(x_{0})}  (1.19)

As shown in Fig. 1.3, the new value x obtained above is a better estimate of the root than either x_{0} or x_{1}, and is used to replace one of them:

• If f(x_{0})f(x_{1}) < 0, then x_{i} (i = 0, 1) satisfying f(x_{i})f(x) > 0 is replaced by x_{2} = x (same as the bisection method);

• If f(x_{0})f(x_{1}) > 0, then x_{i} (i = 0, 1) with greater |f(x_{i})| is replaced by x_{2} = x.

This process is then carried out iteratively to approach the root:

x_{n+1} = x_{n} − \frac{f(x_{n})}{f̂′(x_{n})}  (1.20)

If the derivative f′(x) is available, it can be used to replace its approximation f̂′(x_{n}) and the iteration becomes the Newton-Raphson method (Section 1.4):

x_{n+1} = x_{n} − \frac{f(x_{n})}{f′(x_{n})}  (1.21)

In case f(x_{n}) = f(x_{n−1}), the approximated derivative is zero f̂′(x_{n}) = 0, and x_{n+1} cannot be found. In such a case, a root may exist between x_{n} and x_{n−1}. Therefore the way to resolve this problem is to combine the bisection search with the secant method so that when f̂′ = 0, the algorithm will switch to bisection search. This is Dekker’s method:

x_{n+1} = \{ \begin{matrix}x_{n} − f(x_{n})/f̂′(x_{n}) & if f(x_{n}) ≠ f(x_{n−1}) \\ (x_{n} + x_{n−1})/2 & otherwise \end{matrix}  (1.22)

The order of convergence of the secant method is q = 1.618 as shown in Appendix 1 of this chapter, i.e., the secant method is converges more quickly than the bisection search method with q = 1 (which does not take advantage of the information of the specific function f(x)).

1.3 Fixed-point Iteration

Fixed-point iteration is a method of finding the fixed point of a given function. It can be used to solve the generic equation system f(x) = 0, as seen in this section. In particular, it will be used to solve the Bellman equation in dynamic programming, as we will see in Chapter 20.

We first consider the special case of solving a single-variable nonlinear equation f(x) = 0 (M = N = 1). To find a solution x^{∗} that satisfies the equation, we could first convert it into an equivalent equation g(x) = x, so that an x satisfying one of the equations will also satisfy the other. We then carry out an iteration x_{n+1} = g(x_{n}) from some initial value x_{0}. If the iteration converges at a point x^{∗}, i.e., g(x^{∗}) = x^{∗}, then we also have f(x^{∗}) = 0, i.e., x^{∗} is a solution of the equation f(x) = 0. Consider the following two examples:

Example 1.2

f(x) = log(x) − 0.5 = 0

To solve the equation, we construct another function

g_{1}(x) = x − f(x) = x − log(x) + 0.5

so that the equation g(x) = x is equivalent to the given equation f(x) = 0 (left Fig. 1.4). We then carry out the iteration x_{n+1} = g(x_{n}) from an initial value, such as x_{0} = 0.5, and get

\begin{matrix}n & x_{n} & g(x_{n}) & f(x_{n}) \\ 0 & 0.50000 & 1.69315 & −1.19315 \\ 1 & 1.69315 & 1.66656 & 0.02659 \\ 2 & 1.66656 & 1.65580 & 0.01076 \\ 3 & 1.65580 & 1.65152 & 0.00428 \\ 4 & 1.65152 & 1.64982 & 0.00169 \\ 5 & 1.64982 & 1.64915 & 0.00067 \\ 6 & 1.64916 & 1.64889 & 0.00026 \\ 7 & 1.64889 & 1.64879 & 0.00010 \\ 8 & 1.64879 & 1.64875 & 0.00004 \\ 9 & 1.64875 & 1.64873 & 0.00002 \\ 10 & 1.64873 & 1.64873 & 0.00001 \\ 11 & 1.64873 & 1.64872 & 0.00000 \\ 12 & 1.64872 & 1.64872 & 0.00000 \end{matrix}

We see that the iteration converges to x^{∗} = lim_{n→∞} g(x_{n}) = 1.64872 satisfying g(x^{∗}) = x^{∗} − f(x^{∗}) = x^{∗}, and, equivalently, x^{∗} is also the solution of the given equation f(x) = 0.5 − log(x) = 0:

0.5 − log(1.64872) = 0, i.e., e^{0.5} = \sqrt{e} = 1.64872

Alternatively, we could construct a different function

g′(x) = x + f(x) = x + log(x) − 0.5

also equivalent to f(x) = 0 (right Fig. 1.4). But this iteration no longer converges!

Example 1.3

f(x) = e^{x} − 1/x = 0

Take logarithm of the equation, we get an equivalent form (left in Fig. 1.5)

g(x) = x = e^{−x}

and an iteration:

x_{n+1} = g(x_{n}) = e^{−x_{n}} ⟹^{n→∞} x^{∗} = 0.5671

Figure 1.4 Example 1.2: Iteration converges for g(x) (left) but diverges for g′(x) (right)

Figure 1.5 Example 1.3: Iteration converges for g(x) (left) but diverges for g′(x) (right)

that converges to the root of the given equation f(x) = e^{x} − 1/x = 0, i.e., e^{−0.5671} = 0.5671.

However, if the given equation is convert into an alternative form g′(x) = x = − ln(x) (right in Fig. 1.5), the iteration no longer converges.

Why does this iterative method work in some cases but fail in other? To answer these questions we need to understand the theory behind the method, the fixed point of a contraction function.

If a single variable function g(x) satisfies

|g(x_{1}) − g(x_{2})| ≤ k|x_{1} − x_{2}|, k ≥ 0  (1.23)

it is said to be Lipschitz continuous, and k is a Lipschitz constant. If k = 1, then g(x) is a non-expansive function, if 0 ≤ k < 1, then g(x) is a contraction function or simply a contraction. These concepts can be generalized to solving multivariate equations f(x) = 0.

Definition: In a metric space V with certain distance d(x, y) defined between any two points x, y ∈ V, a function g : V → V is a contraction if

d(g(x), g(y)) ≤ k d(x, y)  (1.24)

The smallest k value that satisfies the above is called the Lipschitz constant.

Intuitively, a contraction reduces the distance between points in the space, i.e., it brings them closer together. A function g(x) may not be a contraction through out its entire domain, but it can be a contraction in the neighborhood of a certain point x^{∗} ∈ V, composed of all points x satisfying d(x, x^{∗}) < ϵ > 0 (i.e., x is a neighbor of x^{∗} as defined by ϵ) so that

d(g(x), g(x^{∗})) ≤ k d(x, x^{∗}), 0 ≤ k < 1  (1.25)

Definition: A fixed point x^{∗} of a function g(x) is a point in its domain that is mapped to itself:

x^{∗} = g(x^{∗})  (1.26)

We immediately have

g(g(x^{∗})) = g^{2}(x^{∗}) = x^{∗}, · · · · · · , g(g(g(⋯ g(x^{∗}) ⋯))) = g^{n}(x^{∗}) = x^{∗} (1.27)

A fixed point x^{∗} is an attractive fixed point if any point x in its neighborhood converges to x^{∗}, i.e., lim_{n→∞} g^{n}(x) = x^{∗}.

We further consider two theorems regarding the fixed point and a contraction mapping.

Fixed-Point Theorem : Let g(x) be a contraction satisfying

d(g(x) − g(y)) ≤ k d(x − y) (0 ≤ k < 1)  (1.28)

then there exists a unique fixed point x^{∗} = g(x^{∗}), which can be found by an iteration from an arbitrary initial point x_{0}:

lim_{n→∞} x_{n} = lim_{n→∞} g(x_{n−1}) = lim_{n→∞} g^{2}(x_{n−2}) = ⋯ = lim_{n→∞} g^{n}(x_{0}) = x^{∗} (1.29)

Contraction Mapping Theorem: Let x^{∗} = g(x^{∗}) be a fixed point of a differentiable function g(x), i.e, ∂g_{i}/∂x_{j} exists for any 1 ≤ i, j ≤ N. If the norm of the Jacobian matrix is smaller than 1, ||J_{g}(x^{∗})|| < 1, then g(x) is a contraction at x^{∗}.

The Jacobian matrix of g(x) is defined as

g′(x) = J_{g}(x) = [ \begin{matrix}\frac{∂g_{1}}{∂x_{1}} & ⋯ & \frac{∂g_{1}}{∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂g_{N}}{∂x_{1}} & ⋯ & \frac{∂g_{N}}{∂x_{N}} \end{matrix} ]  (1.30)

The proofs of the two theorems above are given in Appendix 3 of the chapter. In the special case of a single-variable function g(x) with N = 1, its Jacobian

is simply its derivative g′(x), and we have

g(x) − g(x^{∗}) = g′(x^{∗})(x − x^{∗}) + \frac{1}{2}g′′(x^{∗})(x − x^{∗})^{2} + R(x − x^{∗})  (1.31)

Figure 1.6 Convergence (top) and Divergence (bottom) of the Fixed Point Method

and

|g(x) − g(x^{∗})| ≤ |g′(x^{∗})(x − x^{∗})| ≤ |g′(x^{∗})| |(x − x^{∗})|  (1.32)

If |g′(x)| < 1, then g(x) is a contraction at x^{∗}.

Now we should understand why in the examples of the previous section the iteration leads to convergence in some cases but divergence in other cases: if |g′(x)| < 1, the iteration will converge to the root x^{∗} of f(x) = 0, but if |g′(x)| > 1, it never will never converge.

The iterative process x_{n+1} = g(x_{n}) for finding the fixed of a single-variable function g(x) is shown in Fig. 1.6 as the intersections of the function y = g(x) and the identity function y = x. If |g′(x)| < 1, the iteration converges as shown in the two cases on top, while if |g′(x)| > 1 it diverges as shown in the two cases at the bottom.

In general the fixed-point iteration converges linearly:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|} = lim_{n→∞} \frac{|x_{n+1} − x^{∗}|}{|x_{n} − x^{∗}|} = lim_{n→∞} \frac{|g(x_{n}) − g(x^{∗})|}{|x_{n} − x^{∗}|} = |g′(x^{∗})| = μ < 1

(1.33)

Figure 1.7 Example 1.4, plots of g_{1}(x) (left) and g_{2}(x) (right)

However, if the iteration function g(x) has zero derivative at the fixed point g′(x^{∗}) = 0, we have

g(x) − g(x^{∗}) = g′(x^{∗})(x − x^{∗}) + \frac{1}{2}g′′(x^{∗})(x − x^{∗})^{2} + R(x − x^{∗})
= \frac{1}{2}g′′(x^{∗})(x − x^{∗})^{2} + R(x − x^{∗})  (1.34)

and the iteration converges quadratically:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} = lim_{n→∞} \frac{|x_{n+1} − x^{∗}|}{|x_{n} − x^{∗}|^{2}} = lim_{n→∞} \frac{|g(x_{n}) − g(x^{∗})|}{|x_{n} − x^{∗}|^{2}} = \frac{1}{2}|g′′(x^{∗})| = constant

(1.35)

Moreover, if g′′(x^{∗}) = 0, then the iteration converges cubically.

Example 1.4

f(x) = x^{3} − x − 2 = 0

This equation can converted into an equivalent form of g(x) = x in two different ways:

g_{1}(x) = x = \sqrt[3]{x + 2}, and g_{2}(x) = x = x^{3} − 2

These two functions are plotted in Fig. 1.7, together with f(x) and the identity function. The iteration based on g_{1}(x) (left) converges to the solution x^{∗} = 1.5214 for any initial guess −∞ < x_{0} < ∞, as g_{1}(x) is a contraction. However, the iteration based on g_{2}(x) (right) does not converge to x^{∗} as it is not a contraction in the neighborhood of x^{∗}. In fact, the iteration will diverge towards either −∞ if x_{0} < x^{∗} or ∞ if x_{0} > x^{∗}.

Example 1.5

f(x) = x^{3} − 3x + 1 = 0

we first convert it into the form of g(x) = x in two different ways:

g_{1}(x) = x = \sqrt[3]{3x − 1}, g_{2}(x) = x = \frac{1}{3}(x^{3} − 1)

As can be seen in Fig. 1.8, this equation has three solutions,

x_{1} = −1.8794, x_{2} = 0.3473, x_{3} = 1.5321

of which x_{1} and x_{3} can be obtained by the iteration based on g_{1}(x) and x_{2} can be obtained by the iteration based on g_{2}(x). But neither of them can find all three roots.

• As shown in the plot on the left, |g′_{1}(x)| < 1 for all −∞ < x < ∞ except in the neighborhood of x_{2} = 0.3473, i.e., g_{1}(x) is a contraction mapping everywhere except around x_{2}. Therefore the iteration starting from any initial guess x_{0} will converge to either x_{1} = −1.8749 if x_{0} < x_{2}, or x_{3} = 1.5321 if x_{0} > x_{2}.

x_{n+1} = g_{1}(x_{n}) ⟹^{n→∞} \{ \begin{matrix}x_{1} = −1.8749 & x_{0} < x_{2} \\ x_{3} = 1.5321 & x_{0} > x_{2} \end{matrix}

However, as g_{1}(x) is not a contraction mapping around x = x_{2}, the iteration will never converge to x_{2}.

• As shown in the plot on the right, |g′_{2}(x)| > 1 for all −∞ < x < ∞ except in the neighborhood of x_{2}, i.e., g_{2}(x) is not a contraction mapping around either x_{1} or x_{3}. Therefore the iteration based on g_{2}(x) will not converge to either x_{1} or x_{3}, but it may converge to x_{2}, if the initial guess x_{0} is in the range x_{1} < x_{0} < x_{3}. However, if x_{0} is outside this range the iteration will diverge toward either −∞ if x_{0} < x_{1} of ∞ if x_{0} > x_{3}.

x_{n+1} = g_{2}(x_{n}) ⟹^{n→∞} \{ \begin{matrix}−∞ & x_{0} < x_{1} \\ x_{2} = 0.3473 & x_{1} < x_{0} < x_{3} \\ ∞ & x_{0} > x_{3} \end{matrix}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 1 & −2.0408275e + 00 & −1.3775173e + 00 \\ 2 & −1.9240239e + 00 & −3.5041089e − 01 \\ 3 & −1.8919392e + 00 & −9.6254050e − 02 \\ 4 & −1.8829328e + 00 & −2.7019276e − 02 \\ 5 & −1.8803890e + 00 & −7.6311660e − 03 \\ 6 & −1.8796694e + 00 & −2.1590449e − 03 \\ 7 & −1.8794656e + 00 & −6.1114717e − 04 \\ 8 & −1.8794080e + 00 & −1.7301761e − 04 \\ 9 & −1.8793916e + 00 & −4.8983741e − 05 \\ 10 & −1.8793870e + 00 & −1.3868146e − 05 \\ 11 & −1.8793857e + 00 & −3.9263250e − 06 \\ 12 & −1.8793853e + 00 & −1.1116151e − 06 \end{matrix}

Figure 1.8 Example 1.5, plots of g_{1}(x) (left) and g_{2}(x) (right)

Example 1.6

f(x) = sin(x) − x^{3} = 0

This equation can be converted into the form g(x) = x in different ways: •

g_{0}(x) = x = \sqrt[3]{sin(x)}, g′_{0}(x) = \frac{cos(x)}{3 sin(x)^{2/3}}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 2.000000e + 00 & −7.0907 \\ 1 & 9.688027e − 01 & −0.085089 \\ 2 & 9.375886e − 01 & −0.018075 \\ 3 & 9.306842e − 01 & −0.0041048 \\ 4 & 9.291018e − 01 & −0.00094611 \\ 5 & 9.287364e − 01 & −0.00021881 \\ 6 & 9.286518e − 01 & −5.0647e − 05 \\ 7 & 9.286322e − 01 & −1.1725e − 05 \\ 8 & 9.286277e − 01 & −2.7144e − 06 \end{matrix}

•

g_{1}(x) = x = sin(x)/x^{2}, g′_{1}(x) = \frac{cos(x)}{x^{2}} − \frac{2 sin(x)}{x^{3}}

•

g_{2}(x) = x + sin(x) − x^{3}, g′_{2}(x) = cos(x) − 3 x^{2} + 1

•

g_{3}(x) = x − \frac{sin(x) − x^{3}}{cos(x) − 3x^{2}} , g′_{3}(x) = −\frac{(6x + sin(x))(sin(x) − x^{3})}{(cos(x) − 3x^{2})^{2}}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 2.000000e + 00 & −7.0907 \\ 1 & 1.428913e + 00 & −1.9276 \\ 2 & 1.106787e + 00 & −0.46152 \\ 3 & 9.637850e − 01 & −0.073886 \\ 4 & 9.304466e − 01 & −0.0036294 \\ 5 & 9.286316e − 01 & −1.0509e − 05 \\ 6 & 9.286263e − 01 & −8.9029e − 11 \end{matrix}

Example 1.7

f(x) = e^{x} − \frac{1}{x}

•

g_{0}(x) = x = e^{−x}, g′_{0}(x) = e^{−x}

The iteration from any initial guess x_{0} > 0 will converge to x^{∗} = g_{1}(x^{∗}) = 0.56714.

•

g_{1}(x) = x = − log(x), g′_{1}(x) = 1/x

Around x = x^{∗} = 0.56714, g′_{1}(x) = 1/x > 1, the iteration does not converge.

•

g_{2}(x) = x − \frac{e^{x} − 1/x}{e^{x} + 1/x^{2}} , g′_{2}(x) = \frac{(e^{x} − 1/x)(e^{x} − 2/x^{3})}{(e^{x} + 1/x^{2})^{2}}

Example 1.8 Consider a 3-variable linear vector function f(x) of arguments x = [x, y, z]^{T}:

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = 6x + 3y + 2z − 18 = 0 \\ f_{2}(x) = 2x + 7y + 3z − 25 = 0 \\ f_{3}(x) = x + 3y + 5z − 22 = 0 \end{matrix}

from which the g-function can be obtained:

g(x) = x,  \{ \begin{matrix}g_{1}(x) = x = −(3y + 2z − 18)/6 \\ g_{2}(x) = y = −(2x + 3z − 25)/7 \\ g_{3}(x) = z = −(x + 3y − 22)/5 \end{matrix}

The Jacobian g(x) of this linear system is a constant matrix

J_{g} = [ \begin{matrix}0 & −1/2 & −1/3 \\ −2/7 & 0 & −3/7 \\ −1/5 & −3/5 & 0 \end{matrix} ]

with the induced p=2 norm (maximum singular value) ||J_{g}|| = 0.851 < 1. Consequently, the iteration x_{n+1} = g(x_{n}) converges from an initial guess x_{0} = [1, 1, 1]^{T} to the solution x = [1, 2, 3]^{T}.

Alternatively, the g-function can also be obtained as

g′(x) = x,  \{ \begin{matrix}g′_{1}(x) = x = −(3y + 5z − 22) \\ g′_{2}(x) = y = −(2x + 3z − 25)/7 \\ g′_{3}(x) = z = −(6x + 3y − 18)/2 \end{matrix}

The Jacobian is

J_{g′} = [ \begin{matrix}0 & −3 & −5 \\ −2/7 & 0 & −3/7 \\ −3 & −3/2 & 0 \end{matrix} ]

with the induced p=2 norm ||J_{g}|| = 5.917 > 1. The iteration does not converge.

Example 1.9 Consider a 3-variable nonlinear function f(x) of arguments x = [x, y, z]^{T}:

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = x^{2} − 2x + y^{2} − z + 1 = 0 \\ f_{2}(x) = xy^{2} − x − 3y + yz + 2 = 0 \\ f_{3}(x) = xz^{2} − 3z + yz^{2} + xy = 0 \end{matrix}

The g-function can be obtained as

g(x) = x,  \{ \begin{matrix}g_{1}(x) = x = (x^{2} + y^{2} − z + 1)/2 \\ g_{2}(x) = y = (xy^{2} − x + yz + 2)/3 \\ g_{3}(x) = z = (xz^{2} + yz^{2} + xy)/3 \end{matrix}

With x_{0} = [0, 0, 0]^{T} and after n > 170 iterations x_{n+1} = g(x_{n}) converges to x_{n} = [1.098933, 0.367621, 0.144932]^{T}, with error ||f(x_{n})|| < 10^{−7}. However, the iteration may not converge from other possible initial guesses.

Finally, we consider Aitken’s method, by which the iteration x_{n+1} = g(x_{n}) can be accelerated based on two consecutive points x_{0} and x_{1}, as shown in Fig. 1.9.

The secant line of g(x) that goes through the two points P_{0} = (x_{0}, g(x_{0}) = x_{1}) and P_{1} = (x_{1}, g(x_{1}) = x_{2}) is represented by the equation in terms of its slope:

\frac{y − g(x_{0})}{x − x_{0}} = \frac{g(x_{1}) − g(x_{0})}{x_{1} − x_{0}}  (1.36)

Solving for y, we get

y = g(x_{0}) + (x − x_{0})\frac{g(x_{1}) − g(x_{0})}{x_{1} − x_{0}} = x_{1} + (x − x_{0})\frac{x_{2} − x_{1}}{x_{1} − x_{0}}  (1.37)

To accelerate, instead of moving from x_{0} to x_{1}, we move to the point x at which this secant line intersects with the identity function y = x. We can therefore

Figure 1.9 Aitken’s Method

replace y in the equation above by x and solve the resulting equation

x = x_{1} + (x − x_{0})\frac{x_{2} − x_{1}}{x_{1} − x_{0}}  (1.38)

to get

x = \frac{x_{1}(x_{1} − x_{0}) − x_{0}(x_{2} − x_{1})}{(x_{1} − x_{0}) − (x_{2} − x_{1})} = x_{0} − \frac{(x_{1} − x_{0})^{2}}{x_{2} − 2x_{1} + x_{0}} = x_{0} − \frac{(Δx_{0})^{2}}{Δ^{2}x_{0}} (1.39)

where Δx_{0} and Δ^{2}x_{0} are respectively the first and second order differences defined below:

Δx_{0} = x_{1} − x_{0}, Δx_{1} = x_{2} − x_{1}  (1.40)
Δ^{2}x_{0} = Δx_{1} − Δx_{0} = (x_{2} − x_{1}) − (x_{1} − x_{0}) = x_{2} − 2x_{1} + x_{0}  (1.41)

This result can then be converted into an iterative process

\{ \begin{matrix}x_{n+1} = g(x_{n}) \\ x_{n+2} = g(x_{n+1}) \\ x_{n+3} = x_{n} − (x_{n+1} − x_{n})^{2}/(x_{n+2} − 2x_{n+1} + x_{n}) \end{matrix}  (1.42)

Given x_{n}, we skip x_{n+1} = g(x_{n}) and x_{n+2} = g(x_{n+1}) but directly move to x_{n+3} computed based on x_{n+1} and x_{n+2}, thereby making a greater step towards the solution.

Example 1.10 Solve x^{3} − 3x + 1 = 0. Construct g(x) = (3x − 1)^{1/3}. It takes 18 iterations for the regular fixed-point algorithm with initial guess x_{0} = 1, to get x_{18} = 1.5320887 that satisfies |f(x_{n})| < 10^{−6}, but it only three iterations for Aitken’s method to converge to the same result:

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 1.0000 & −1.0000 \\ 1 & 1.259921 & −7.797631e − 01 \\ 2 & 1.406056 & −4.384047e − 01 \\ 3 & 1.476396 & −2.110206e − 01 \\ 4 & 1.507985 & −9.476760e − 02 \\ 5 & 1.521751 & −4.129594e − 02 \\ 6 & 1.527672 & 1.776362e − 02 \\ 7 & 1.530205 & −7.598914e − 03 \\ 8 & 1.531286 & 3.242990e − 03 \\ 9 & 1.531747 & −1.382619e − 03 \\ 10 & 1.531943 & −5.892137e − 04 \\ 11 & 1.532027 & −2.510521e − 04 \\ 12 & 1.532062 & −1.069599e − 04 \\ 13 & 1.532078 & −4.556840e − 05 \\ 14 & 1.532084 & −1.941335e − 05 \\ 15 & 1.532087 & −8.270551e − 06 \\ 16 & 1.532088 & −3.523444e − 06 \\ 17 & 1.532089 & −1.501066e − 06 \\ 18 & 1.532089 & −6.394874e − 07 \end{matrix}

\begin{matrix}n & x_{n} & f(x_{n}) \\ 0 & 1.0000 & −1.0000 \\ 1 & 1.5937361 & 2.668730e − 01 \\ 2 & 1.5323992 & 1.254749e − 03 \\ 3 & 1.5320889 & 3.421160e − 08 \end{matrix}

1.4 Newton-Raphson Method (Univariate)

The Newton-Raphson method, or simply Newton’s method, solves the general equation f(x) = 0 more efficiently as its converges more quickly (quadratic) than the secant method previously discussed, due to the derivative f′(x) assumed to be available, as an extra piece of information in additional to the function value f(x) itself. Newton’s method is based on the approximation of the function f(x) by its Taylor series expansion at any x_{0} as an initial guess of the root:

f(x) = f(x_{0})+f′(x_{0})(x−x_{0})+ \frac{1}{2!}f′′(x)(x−x_{0})^{2} +⋯+ \frac{1}{n!}f^{(n)}(x_{0})(x−x_{0})^{n} +⋯

(1.43)

Figure 1.10 Newton-Raphson Method

If f(x) is linear, i.e., its slope f′(x) is a constant at any point x, then all terms other than the first two are zero, and the equation becomes

f(x) = f(x_{0}) + f′(x)(x − x_{0}) = 0  (1.44)

Solving this equation for x we get the root x^{∗}:

x^{∗} = x_{0} − \frac{f(x_{0})}{f′(x_{0})} = x_{0} + Δx_{0}  (1.45)

where Δx_{0} = −f(x_{0})/f′(x_{0}) is the step increment needed to move from the initial guess x_{0} to the root x^{∗}:

Δx_{0} = −\frac{f(x_{0})}{f′(x_{0})} \{ \begin{matrix}< 0 & if f(x_{0}) and f′(x_{0}) are of the same sign \\ > 0 & if f(x_{0}) and f′(x_{0}) are of different signs \end{matrix}  (1.46)

If f(x) is nonlinear, the sum of the first two terms of its Taylor series is only an approximation of the function, and the result in Eq. (1.45) is no longer the root, but an approximation of the root, which can be improved iteratively to gradually approach the root x^{∗}:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − \frac{f(x_{n})}{f′(x_{n})}, n = 0, 1, 2, ⋯  (1.47)

as illustrated in Fig. 1.10. Note that this iteration is the same as Eq. (1.20) for the secant method, with f̂′(x_{n}) replaced by the derivative f′(x_{n}).

The Newton-Raphson method can be considered as the fixed-point iteration x_{n+1} = g(x_{n}) based on

g(x) = x − f(x)/f′(x)  (1.48)

based on the assumption that f′(x) ≠ 0. The root x^{∗} at which f(x^{∗}) = 0 is also the fixed point of g(x), i.e., g(x^{∗}) = x^{∗}. For the iteration to converge, g(x) needs

Figure 1.11 Oscillation

to be a contraction with |g′(x)| < 1. Consider

g′(x) = (x − \frac{f(x)}{f′(x)})′ = 1 − \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.49)

At the root x = x^{∗} where f(x^{∗}) = 0, if f′(x^{∗}) ≠ 0, then g′(x^{∗}) = 0 < 1, i.e., g(x) is a contraction and the iteration x_{n+1} = g(x_{n}) converges quadratically when x_{n} is close enough to x^{∗}.

Here are some additional considerations of the Newton-Raphson methods:

• If f′(x_{n}) = 0 (with a horizontal tangent line), the iteration cannot proceed, but we can modify x_{n} by adding a small value ϵ to x_{n} so that f′(x+ϵ) ≠ 0.

• It is difficult to know the number of roots of a nonlinear equation (unlike the case of a linear equation), which can vary from zero (e.g., f(x) = x^{2} + 1) to infinity (e.g., f(x) = cos(x)). One can try different initial guesses in the range of interest to see if different roots can be found.

• Sometime a parameter δ can be used to control the step size of the iteration:

x_{n+1} = x_{n} − δ \frac{f(x_{n})}{f′(x_{n})}  (1.50)

– If δ < 1, the iteration is de-accelerated. Although the convergence becomes slower, this may be desirable if the function f(x) is not smooth with many local variations.

– If δ > 1, the iteration is accelerated. The convergence may or may not be accelerated. Due to the greater step size, the root may be skipped and missed. Sometimes the convergence may become significantly slowed or even oscillate around the true root, such as the example shown in Fig. 1.11 with δ = 2.

The Newton-Raphson method converges quadratically:

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} ≤ μ  (1.51)

as shown in Appendix 2 of this chapter, i.e., it converges more quickly than both the bisection method (q = 1) and the secant method (q = 1.618).

Example 1.11

f(x) = x^{3} − 4x^{2} + 5x − 2 = (x − 1)^{2}(x − 2) = 0  (1.52)

Obviously this equation has a repeated root x = 1 as well as a single root x = 2. We have

f′(x) = 3x^{2} − 8x + 5 = (3x − 5)(x − 1),  f′′(x) = 6x − 8  (1.53)

Note that at the root x^{∗} = 1 we have f′(x^{∗}) = f′(1) = 0. We further find:

g(x) = x − \frac{f(x)}{f′(x)} = x − \frac{(x − 1)^{2}(x − 2)}{(3x − 5)(x − 1)} = x − \frac{(x − 1)(x − 2)}{3x − 5}  (1.54)

and

g′(x) = \frac{f(x)f′′(x)}{(f′(x))^{2}} = \frac{(x − 1)^{2}(x − 2)(6x − 8)}{(3x − 5)^{2}(x − 1)^{2}}
= \frac{(x − 2)(6x − 8)}{(3x − 5)^{2}} = \{ \begin{matrix}1/2 & x = 1 \\ 0 & x = 2 \end{matrix}

We therefore have

e_{n+1} = g′(x)e_{n} − \frac{1}{2}g′′(x)e_{n}^{2} + O(e_{n}^{3}) ⟹^{n→∞} \{ \begin{matrix}e_{n}/2 & x = 1 \\ −g′′(x)e_{n}^{2}/2 & x = 2 \end{matrix}  (1.55)

We see that the iteration converges quadratically to the single root x = 2, but only linearly to the repeated root x = 1.

We consider in general a function with a repeated root at x = a of multiplicity k:

f(x) = (x − a)^{k}h(x)  (1.56)

its derivative is

f′(x) = k(x−a)^{k−1}h(x)+(x−a)^{k}h′(x) = (x−a)^{k−1}[kh(x)+(x−a)h′(x)] (1.57)

As f′(a) = 0, the convergence of the Newton-Raphson method to x = a is linear, rather than quadratic.

In such case, we can accelerate the iteration by using a step size δ = k > 1:

g(x) = x − k \frac{f(x)}{f′(x)}  (1.58)

and

g′(x) = 1 − k \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = 1 − k + k \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.59)

Now we show that this g′(x) is zero at the repeated root x = a, therefore the convergence to this root is still quadratic.

We substitute f(x) = (x − a)^{k}h(x), f′(x) = k(x − a)^{k−1}[kh + (x − a)h′], and

f′′(x) = [(x − a)^{k−1}[kh + (x − a)h′]]′
= 1 − k + k (k − 1)(x − a)^{k−2}[kh + (x − a)h′]
+(x − a)^{k−1}[(k + 1)h′ + (x − a)h′′]  (1.60)

into the expression for g′(x) above to get (after some algebra):

g′(x) = 1 − k + k \frac{f(x)f′′(x)}{(f′(x))^{2}}
= 1 − k + k \frac{h(k − 1)(kh + (x − a)h′) + h(x − a)((k + 1)h′ + (x − a)h′)}{(kh + (x − a)h′)^{2}} (1.61)

At x = a, we get

g′(x)|_{x=a} = 1 − k + k \frac{(k − 1)kh^{2}}{(kh)^{2}} = 0  (1.62)

i.e., the convergence to the repeated root at x = a is no longer linear but quadratic. The difficulty, however, is that the multiplicity k of a root is unknown ahead of time. If δ > 1 is used blindly some root may be skipped, and the iteration may oscillate around the real root.

1.5 Newton-Raphson Method (Multivariate)

The Newton-Raphson method discussed above for solving a single-variable equation f(x) = 0 can be generalized to the case of multivariate equation systems containing M equations of N variables in x = [x_{1}, ⋯, x_{N}]^{T}:

f(x) = [ f_{1}(x) ; ⋮ ; f_{M}(x) ] = [ f_{1}(x_{1}, ⋯, x_{N}) ; ⋮ ; f_{M}(x_{1}, ⋯, x_{N}) ] = [ 0 ⋮ 0 ] = 0  (1.63)

Again, we first consider the Taylor series expansion (Eq. (A.391)) of each of the M functions in the eighborhood of an initial guess x_{0} = [x_{01}, ⋯, x_{0N}]^{T}:

f_{m}(x) = f_{m}(x_{0}) + g_{f_{m}}^{T}(x_{0})(x − x_{0}) + r_{m}(||x − x_{0}||^{2})
≈ f_{m}(x_{0}) + ∑_{n=1}^{N} \frac{∂f_{m}(x_{0})}{∂x_{n}}(x_{n} − x_{0n})  (m = 1, ⋯, M) (1.64)

where r_{m}(||x − x_{0}||^{2}) represents the second and higher order terms in the series beyond the linear term, which can be neglected if ||x − x_{0}|| is small. These M equations can be expressed in matrix form

[ f_{1}(x) ; ⋮ ; f_{M}(x) ] ≈ [ f_{1}(x_{0}) ; ⋮ ; f_{M}(x_{0}) ] + [ \begin{matrix}\frac{∂f_{1}(x_{0})}{∂x_{1}} & ⋯ & \frac{∂f_{1}(x_{0})}{∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{M} (x_{0})}{∂x_{1}} & ⋯ & \frac{∂f_{M} (x_{0})}{∂x_{N}} \end{matrix} ] [ x_{1} − x_{01} ; ⋮ ; x_{N} − x_{0N} ]

(1.65)

or more concisely

f(x) ≈ f(x_{0}) + J(x_{0}) (x − x_{0}) = f_{0} + J_{0} Δx  (1.66)

where Δx = x − x_{0}, and f_{0} = f(x_{0}) and J_{0} = J(x_{0}) are respectively the function f(x) and its Jacobian matrix J_{f}(x) both evaluated at x_{0}. We further consider solving the equation system f(x) = 0 in the following two cases:

• M = N: The number of equations is the same as the number of unknowns, the Jacobian J(x) is a square matrix and its inverse J^{−1} exists in general. In the special case where f(x) = Ax − b is linear, the Taylor series contains only the first two terms while all higher order terms are zero, and the approximation in Eq. (1.66) becomes exact with J = A. To find the root x^{∗} of the equation, we set f(x) in the equation to zero and solve the resulting equation for x to get the solution:

x^{∗} = x_{0} − J_{0}^{−1}f_{0} = x_{0} − A^{−1}f_{0}  (1.67)

For example, if x_{0} = 0 and f(x_{0}) = −b, then x^{∗} = A^{−1}b.

If f(x) is nonlinear, the sum of the first two terms of the Taylor series is only an approximation, and the result in Eq. (1.67) is only an approximate root, which can be further improved iteratively to move from the initial guess x_{0} towards the root x^{∗}:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − J_{n}^{−1} f_{n}  (1.68)

where Δx_{n} = −J_{n}^{−1} f_{n} is the increment, representing the search direction at the nth step. The iteration moves x_{n} in the N-D space spanned by {x_{1}, ⋯, x_{N}} from some initial guess x_{0} along such a path that all function values f_{m}(x), m = 1, ⋯, M) are reduced. As in the univariate case, a scaling factor δ_{n} can be used to control the step size of the iteration

x_{n+1} = x_{n} + δ_{n} Δx_{n} = x_{n} − δ_{n} J_{n}^{−1} f_{n}  (1.69)

The step size can be adjusted based on the specific function shape. When δ_{n} < 1, the step size becomes smaller and the convergence of the iteration is slower, we will have a better chance not to skip a solution, which may happen if f(x) is not smooth and the step size is too big.

The algorithm is listed below (where the tolerance tol is preset to a small value):

– Select x_{0}

– Obtain f_{0} = f(x_{0}) and J_{0}

– Obtain J_{0}^{−1}

– n = 0

– While ||x_{n+1} − x_{n}|| > tol do ∘ x_{n+1} = x_{n} − J_{n}^{−1}f_{n} ∘ Find f_{n+1} and J_{n+1} ∘ n = n + 1

• M > N: There are more equations than unknowns, i.e., equation f(x) = 0 is an over-constrained system, and the Jacobian J(x) is an M ×N non-square matrix with no inverse, i.e., no solution exists for the equation f(x) = 0 in general. But we can still seek to find an optimal solution x^{∗} that minimizes the following sum-of-squares error (SSE):

ε(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2}f(x)^{T} f(x) = \frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)  (1.70)

The gradient vector of this SSE is

g_{ε}(x) = \frac{d}{dx}ε(x) = \frac{d}{dx} (\frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)) = ∑_{m=1}^{M} \frac{d f_{m}(x)}{dx}f_{m}(x)
= J_{f}^{T} (x) f(x)  (1.71)

where J_{f}(x) is the Jacobian of f(x). We can also obtain the nth component (n = 1, ⋯, N) of g_{ε}(x) as:

\frac{∂ε(x)}{∂x_{n}} = ∑_{m=1}^{M} \frac{∂f_{m}(x)}{∂x_{n}} f_{m}(x) = ∑_{m=1}^{M} J_{mn} f_{m}(x)  (1.72)

where J_{mn} = ∂f_{m}(x)/∂x_{n} is the component in the mth row and nth column of J_{f}(x). At the optimal solution x^{∗} that minimizes this error ε(x), we have g_{ε}(x^{∗}) = 0.

In the special case when the function f(x) = Ax − b is linear with J(x) = A, it can be represented as the sum of the first two terms of its Taylor series in Eq. (1.66), and the gradient is:

g_{ε}(x) = A^{T} f(x) = A^{T} [f(x_{0}) + A Δx] = A^{T} (f_{0} + AΔx)  (1.73)

where x_{0} is any chosen initial guess. At the optimal solution x^{∗} that minimizes ε(x), this gradient is zero:

g_{ε}(x^{∗}) = A^{T} (f_{0} + AΔx) = 0  (1.74)

Solving this equation for Δx we get the optimal increment

Δx = −(A^{T} A)^{−1}A^{T} f_{0} = −A^{−} f_{0}  (1.75)

Here A^{−} = (A^{T} A)^{−1}A^{T} is the pseudo-inverse (Section A.6.1) of the nonsquare matrix A. Now the optimal solution can be found as:

x^{∗} = x_{0} + Δx = x_{0} − A^{−} f_{0}  (1.76)

For example, if x_{0} = 0 and f(x_{0}) = Ax_{0} − b = −b, we have x^{∗} = A^{−}b, same as Eq. (1.8).

In the general case when f(x) is nonlinear, the sum of the first two terms of its Taylor series is its approximation, and the result above is an approximation of the root, which can be further improved iteratively to move from the initial guess towards the root:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − J_{n}^{−} f_{n}  (1.77)

This iteration will converge to x^{∗} at which g_{ε}(x^{∗}) = 0, and the squared error ε(x) is minimized.

Comparing Eqs. (1.68) and (1.77), we see that they are essentially the same, with the only difference that the regular inverse J^{−1} is used when M = N, but the pseudoinverse J^{−} is used when M > N and J^{−1} does not exist.

The Newton-Raphson method assumes the availability of the analytical expressions of all partial derivatives J_{mn} = ∂f_{m}(x)/∂x_{n} (m = 1, ⋯, M, n = 1, ⋯, N) in the Jacobian matrix J. However, when this is not the case, J_{mn} need to be approximated by forward or central difference (secant) method:

J_{mn} = \frac{∂f_{m}(x_{1}, ⋯, x_{N})}{∂x_{n}} ≈ \frac{f_{m}(x_{1}, ⋯, x_{n} + h, ⋯, x_{N}) − f_{m}(x_{1}, ⋯, x_{n}, ⋯, x_{N})}{h}
≈ \frac{f_{m}(x_{1}, ⋯, x_{n} + h, ⋯, x_{N}) − f_{m}(x_{1}, ⋯, x_{n} − h, ⋯, x_{N})}{2h}  (1.78)

where h is a small increment.

Example 1.12

\{ \begin{matrix}3 x_{1} − cos(x_{2}x_{3}) − 3/2 = 0 \\ 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{3} − 1 = 0 \\ 20 x_{3} + e^{−x_{1}x_{2}} + 9 = 0 \end{matrix}

J = [ \begin{matrix}3 & x_{3} sin(x_{2}x_{3}) & x_{2} sin(x_{2}x_{3}) \\ 8 x_{1} & −1250 x_{2} & 2 \\ −x_{2} e^{−x_{1}x_{2}} & −x_{1} e^{−x_{1}x_{2}} & 20 \end{matrix} ]

\begin{matrix}n & x & error \\ 0 & (1.000000, 1.000000, 1.000000) & 6.207e + 02 \\ 1 & (1.232701, 0.503132, −0.473253) & 1.541e + 02 \\ 2 & (0.832592, 0.251806, −0.490636) & 3.884e + 01 \\ 3 & (0.833238, 0.128406, −0.494702) & 9.517e + 00 \\ 4 & (0.833275, 0.069082, −0.497147) & 2.200e + 00 \\ 5 & (0.833281, 0.043585, −0.498206) & 4.063e − 01 \\ 6 & (0.833282, 0.036117, −0.498517) & 3.486e − 02 \\ 7 & (0.833282, 0.035343, −0.498549) & 3.741e − 04 \\ 8 & (0.833282, 0.035335, −0.498549) & 4.498e − 08 \\ 9 & (0.833282, 0.035335, −0.498549) & 5.551e − 16 \end{matrix}

Example 1.13

f(x) = 0,  \{ \begin{matrix}f_{1}(x) = x_{1}^{2} − 2x_{1} + x_{2}^{2} − x_{3} + 1 = 0 \\ f_{2}(x) = x_{1}x_{2}^{2} − x_{1} − 3x_{2} + x_{2}x_{3} + 2 = 0 \\ f_{3}(x) = x_{1}x_{3}^{2} − 3x_{3} + x_{2}x_{3}^{2} + x_{1}x_{2} = 0 \end{matrix}

J = [ \begin{matrix}2x_{1} − 2 & 2x_{2} & −1 \\ x_{2}^{2} − 1 & 2x_{1}x_{2} − 3 + x_{3} & x_{2} \\ x_{3}^{2} + x_{2} & x_{3}^{2} + x_{1} & 2x_{1}x_{3} − 3 + 2x_{2}x_{3} \end{matrix} ]

With x_{0} = [1, 2, 3]^{T}, we get a root:

\begin{matrix}n & x & error \\ 0 & (1.00000, 2.00000, 3.00000) & 2.064e + 01 \\ 1 & (0.10256, 1.64103, 2.56410) & 4.303e + 00 \\ 2 & (1.52062, 1.41113, 0.19859) & 2.689e + 00 \\ 3 & (1.94123, 0.77134, 0.89465) & 1.217e + 00 \\ 4 & (1.06737, 1.19117, 0.48353) & 1.144e + 00 \\ 5 & (1.26825, 0.95182, 0.88028) & 3.323e − 01 \\ 6 & (0.95899, 1.03384, 0.96813) & 1.171e − 01 \\ 7 & (1.00171, 1.00007, 0.99718) & 4.162e − 03 \\ 8 & (1.00000, 1.00000, 1.00000) & 6.701e − 06 \end{matrix}

With x_{0} = [0, 0, 0]^{T}, we get another root:

\begin{matrix}n & x & error \\ 0 & (0.00000, 0.00000, 0.00000) & 2.236e + 00 \\ 1 & (0.50000, 0.50000, 0.00000) & 5.728e − 01 \\ 2 & (0.83951, 0.47531, 0.13580) & 1.175e − 01 \\ 3 & (0.98582, 0.41849, 0.15069) & 2.639e − 02 \\ 4 & (1.05417, 0.38715, 0.14717) & 6.088e − 03 \\ 5 & (1.08565, 0.37339, 0.14558) & 1.264e − 03 \\ 6 & (1.09693, 0.36849, 0.14503) & 1.618e − 04 \\ 7 & (1.09888, 0.36764, 0.14494) & 4.817e − 06 \end{matrix}

Broyden’s method

In the Newton-Raphson method, two main operations are carried out in each iteration: (a) evaluate the Jacobian matrix J_{f} (x_{n}) and (b) obtain its inverse J_{f}^{−1}(x_{n}). To avoid the expensive computation for these operations, we can consider using Broyden’s method, one of the so called quasi-Newton methods, which approximates the inverse of the Jacobian J_{n+1}^{−1} = J^{−1}(x_{n+1}) from the J_{n}^{−1} = J^{−1}(x_{n}) in the previous iteration step, so that it can be updated iteratively from the initial J_{0}^{−1} = J^{−1}(x_{0}).

We first consider the iteration step in the single-variable case to see how to estimate the next derivative f′_{n+1} = f′(x_{n+1}) from the current f′_{n} = f′(x_{n}) by the secant method:

f′_{n+1} ≈ f̂′_{n+1} = \frac{f_{n+1} − f_{n}}{x_{n+1} − x_{n}} = \frac{δf_{n}}{δx_{n}} = \frac{f̂′_{n}δx_{n} − f̂′_{n}δx_{n} + δf_{n}}{δx_{n}}
= f̂′_{n} + \frac{δf_{n} − f̂′_{n}δx_{n}}{δx_{n}} = f̂′_{n} + δ̂f′_{n}  (1.79)

where

• δf_{n} = f_{n+1} − f_{n} is the true increment of the function over the interval δx_{n} = x_{n+1} − x_{n};

• f̂′_{n}δx_{n} is the estimated increment of the function based on the previous derivative f̂′_{n};

• δ̂f′_{n} is the estimated increment of the derivative:

δ̂f′_{n} = \frac{δf_{n} − f̂′_{n}δx_{n}}{δx_{n}}  (1.80)

The equation above indicates that the derivative f′_{n+1} in the (n + 1)th step can be estimated by adding the estimated increment δ̂f′_{n} to the derivative f̂′n in the current nth step, as shown in Fig. refBroyden1.

Having obtained f̂′_{n+1}, we can use the same iteration in the Newton-Raphson

Figure 1.12 Broyden’s Method (Section 1.5)

method to find x_{n+1}:

x_{n+1} = x_{n} − \frac{f(x_{n})}{f̂′_{n}}  (1.81)

This method for single-variable case can be generalized to multiple variable case for solving f(x) = 0. Following the way we estimate the increment of the derivative of a single-variable function in Eq. (1.80), here we can estimate the increment of the Jacobian of a multi-variable function:

δĴ_{n} = \frac{(δf_{n} − Ĵ_{n}δx_{n}) δx_{n}^{T}}{δx_{n}^{T} δx_{n}} = \frac{(δf_{n} − Ĵ_{n}δx_{n})δx_{n}^{T}}{||δx_{n}||^{2}}  (1.82)

where δx_{n} = x_{n+1} − x_{n} and δf_{n} = f_{n+1} − f_{n} = f(x_{n+1}) − f(x_{n}). Now in each iteration, we can update the estimated Jacobian as well as the estimated root:

Ĵ_{n+1} = Ĵ_{n} + δĴ_{n},  x_{n+1} = x_{n} + δx_{n} = x_{n} − Ĵ_{n}^{−1}f(x_{n})  (1.83)

The algorithm can be further improved so that the inverse of the Jacobian J_{n} is avoided. Specifically, consider the inverse Jacobian:

Ĵ_{n+1}^{−1} = (Ĵ_{n} + δĴ_{n})^{−1} = [Ĵ_{n} + \frac{(δf_{n} − Ĵ_{n}δx_{n})δx_{n}^{T}}{||δx_{n}||^{2}} ]^{−1}  (1.84)

We can apply the Sherman-Morrison formula (Section A.2.4):

(A + uv^{T})^{−1} = A^{−1} − \frac{A^{−1}uv^{T} A^{−1}}{1 + v^{T} A^{−1}u}  (1.85)

to the right-hand side of the equation above by defining

A = Ĵ_{n},  u = \frac{δf_{n} − Ĵ_{n}δx_{n}}{||δx_{n}||^{2}} ,  v = δx_{n}  (1.86)

and rewrite it as:

Ĵ_{n+1}^{−1} = Ĵ_{n}^{−1} − \frac{Ĵ_{n}^{−1} (δf_{n} − Ĵ_{n}δx_{n}^{T})/||δx_{n}||^{2} δx_{n}^{T} Ĵ_{n}^{−1}}{1 + δx_{n}^{T} Ĵ_{n}^{−1}(δf_{n} − Ĵ_{n}δx_{n})/||δx_{n}||^{2}} = Ĵ_{n}^{−1} − \frac{(Ĵ_{n}^{−1}δf_{n} − δx_{n}) δx_{n}^{T} Ĵ_{n}^{−1}}{δx_{n}^{T} Ĵ_{n}^{−1}δf_{n}}

(1.87)

We see that the next Ĵ_{n+1}^{−1} can be iteratively estimated directly from the previous Ĵ_{n}^{−1}, thereby avoiding computing the inverse of Ĵ_{n} altogether. The algorithm is listed below:

• Select x_{0}

• Find f_{0} = f(x_{0}), J_{0} = J(x_{0}), and J_{0}^{−1}

• x_{1} = x_{0} − J_{0}^{−1}f_{0}

• n = 0

• While ||x_{n+1} − x_{n}|| > tol do – δx_{n} = x_{n+1} − x_{n}

– f_{n+1} = f(x_{n+1})

– δf = f_{n+1} − f_{n}

– J_{n}^{−1} = J_{n}^{−1} − (δx_{n} − J_{n}^{−1}δf_{n})δx_{n}^{T} J_{n}^{−1}/(δx_{n}^{T} J_{n}^{−1}δf_{n})

– x_{n} = x_{n+1}, f_{n} = f_{n+1}

– x_{n+1} = x_{n} − J_{n}^{−1}f_{n}

– n = n + 1

Problems

Develop Matlab code (or any other language) to implement the algorithms covered in this chapter to solve the equations in the examples in the text. Try a set of different initial guesses to see how they may affect how quickly the iteration converges.

1. Implement the bisection and secant methods and apply them to solve the single-variable equations considered in Examples 1.2 through 1.6 based on properly selected initial guesses.

2. Implement method of fixed-point iteration and apply it to solve the same single-variable equations in the previous problem based on properly selected initial guesses.

3. Implement the univariate Newton-Raphson method and apply it to solve the same single-variable equations above.

4. Implement the multivariate Newton-Raphson method and apply it to solve the multivariable equations in Examples 1.12 and 1.13.

Appendices

1.A Order of Convergence of the Secant Method

Let x^{∗} be the root at which f(x^{∗}) = 0. The error of x_{n+1} is:

e_{n+1} = x_{n+1} − x^{∗} = x_{n} − \frac{x_{n} − x_{n−1}}{f(x_{n}) − f(x_{n−1})} f(x_{n}) − x^{∗}
= \frac{(x_{n−1} − x^{∗})f(x_{n}) − (x_{n} − x^{∗})f(x_{n−1})}{f(x_{n}) − f(x_{n−1})}
= \frac{e_{n−1}f(x_{n}) − e_{n}f(x_{n−1})}{f(x_{n}) − f(x_{n−1})}  (1.88)

Consider the Taylor expansion of f(x_{n}) around the root at which as f(x^{∗}) = 0:

f(x_{n}) = f(x^{∗} + e_{n}) = f(x^{∗}) + f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})
= f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})  (1.89)

Similarly we also have

f(x_{n−1}) = f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})  (1.90)

Substituting these into the expression for e_{n+1} above we get

e_{n+1} = \frac{e_{n−1}[f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})] − e_{n}[f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})]}{[f′(x^{∗})e_{n} + \frac{1}{2}f′′(x^{∗})e_{n}^{2} + O(e_{n}^{3})] − [f′(x^{∗})e_{n−1} + \frac{1}{2}f′′(x^{∗})e_{n−1}^{2} + O(e_{n−1}^{3})]}
= \frac{e_{n−1}e_{n}f′′(x^{∗})(e_{n} − e_{n−1})/2 + O(e_{n−1}^{4})}{(e_{n} − e_{n−1})f′(x^{∗}) + (e_{n}^{2} − e_{n+1}^{2})f′′(x^{∗})/2 + O(e_{n−1}^{3})}
= \frac{e_{n−1}e_{n}f′′(x^{∗})/2 + O(e_{n−1}^{3})}{f′(x^{∗}) + (e_{n} + e_{n+1})f′′(x^{∗})/2 + O(e_{n−1}^{2})}
= \frac{e_{n−1}e_{n}f′′(x^{∗})/2 + O(e_{n−1}^{3})}{f′(x^{∗}) + O(e_{n}) + O(e_{n−1}^{2})}  (1.91)

When n → ∞, the lowest order terms in both the numerator and denominator become the dominant terms as all other higher order terms approach to zero, and we have

e_{n+1} = \frac{e_{n−1}e_{n}f′′(x^{∗})}{2f′(x^{∗})} = C e_{n}e_{n−1}  (1.92)

where we have defined a constant C = f′′(x^{∗})/2f′(x^{∗}). To find the order of convergence, we need to find q in

|e_{n+1}| ≤ |C| |e_{n−1}| |e_{n}| = μ|e_{n}|^{q}  (1.93)

Solving this equation for |e_{n}| we get

|e_{n}| = (\frac{|C|}{μ}|e_{n−1}|)^{1/(q−1)} = (\frac{|C|}{μ})^{1/(q−1)} |e_{n−1}|^{1/(q−1)}  (1.94)

On the other hand, when n → ∞ we also have

|e_{n}| = μ|e_{n−1}|^{q}  (1.95)

Equating the right-hand sides of the two equations above we get

(\frac{|C|}{μ})^{1/(q−1)} |e_{n−1}|^{1/(q−1)} = μ|e_{n−1}|^{q}  (1.96)

which requires the following two equations to hold:

q = \frac{1}{q − 1},  μ = (\frac{|C|}{μ})^{1/(q−1)} = (\frac{|C|}{μ})^{q}  (1.97)

These two equations can be solved separately to get

q = \frac{1 + \sqrt{5}}{2} = 1.618, μ = |C|^{q/(q+1)} = C^{(\sqrt{5}−1)/2} = |C|^{0.618}  (1.98)

i.e.,

|e_{n+1}| = μ|e_{n}|^{q} = |C|^{0.618}|e_{n}|^{1.618} = |\frac{f′′(x)}{2f′(x)}|^{0.618}|e_{n}|^{1.618}  (1.99)

or

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{1.618}} = μ = |C|^{0.618}  (1.100)

We see that the secant method has an order of convergence q = 1.618 with rate of convergence μ = |C|^{0.618}.

1.B Order of Convergence of the Newton-Raphson Method

The order of convergence of the Newton-Raphson iteration can be found based on the Taylor expansion of f(x) at the neighborhood of the root x^{∗} = x_{n} + e_{n} (Section A.7):

0 = f(x^{∗}) = f(x_{n}) + f′(x_{n})e_{n} + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.101)

where e_{n} = x^{∗} − x_{n} is the error at the nth step. Substituting the Newton-Raphson’s iteration

x_{n+1} = x_{n} − \frac{f(x_{n})}{f′(x_{n})}, i.e. f(x_{n}) = f′(x_{n})(x_{n} − x_{n+1})  (1.102)

into the equation above, we get

0 = f′(x_{n})(x_{n} − x_{n+1}) + f′(x_{n})(x^{∗} − x_{n}) + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})
= f′(x_{n})(x^{∗} − x_{n+1}) + \frac{f′′(x)}{2}e_{n}^{2} + O(e_{n}^{3})
= f′(x_{n})e_{n+1} + \frac{f′′(x_{n})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.103)

i.e.

e_{n+1} = −\frac{f′′(x_{n})}{2f′(x_{n})}e_{n}^{2} + O(e_{n}^{3})  (1.104)

When n → ∞ all the higher order terms disappear, and the above can be written as

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} ≤ \frac{|f′′(x^{∗})|}{2|f′(x^{∗})|} = μ  (1.105)

Alternatively, we can get the Taylor expansion in terms of g(x):

x_{n+1} = x^{∗} − e_{n+1} = g(x_{n}) = g(x^{∗} − e_{n}) = g(x^{∗}) − g′(x^{∗})e_{n} + \frac{g′′(x^{∗})}{2}e_{n}^{2} + O(e_{n}^{3})

(1.106)

Subtracting g(x^{∗}) = x^{∗} from both sides we get:

e_{n+1} = g′(x^{∗})e_{n} − \frac{g′′(x^{∗})}{2}e_{n}^{2} + O(e_{n}^{3})  (1.107)

Now we find g′(x) and g′′(x):

g′(x) = (x − \frac{f(x)}{f′(x)})′ = 1 − \frac{(f′(x))^{2} − f(x)f′′(x)}{(f′(x))^{2}} = \frac{f(x)f′′(x)}{(f′(x))^{2}}  (1.108)

and

g′′(x) = (\frac{f(x)f′′(x)}{(f′(x))^{2}})′ = \frac{(f′(x)f′′(x) + f(x)f′′′(x))(f′(x))^{2} − f(x)f′′(x)2f′(x)f′′(x)}{(f′(x))^{4}}
= \frac{(f′(x))^{3}f′′(x)}{(f′(x))^{4}} = \frac{f′′(x)}{f′(x)}  (1.109)

Evaluating these at x = x^{∗} at which f(x^{∗}) = 0, and substituting them back into the expression for e_{n+1} above, we see that the linear term is zero as g′(x^{∗}) = 0, and get the same result as in Eq. (1.105):

lim_{n→∞} \frac{|e_{n+1}|}{|e_{n}|^{2}} ≤ \frac{|f′′(x^{∗})|}{2|f′(x^{∗})|} = \frac{|g′′(x^{∗})|}{2}  (1.110)

We see that if f′(x^{∗}) ≠ 0, the order of convergence of the Newton-Raphson method is q = 2 with the rate of convergence μ = |f′′(x)|/2|f′(x)|. However, if f′(x^{∗}) = 0, the convergence is linear rather than quadratic, as shown in some of the examples.

1.C Proofs of the Fixed Point and Contraction Mapping Theorems

Proof of the fixed point theorem:

Here the distance d(x − y) is specifically defined as the p-norm (section A.4.1) of the vector x − y:

d(x, y) = ||x − y||_{p} = (∑_{n=1}^{N}(x_{n} − y_{n})^{p})^{1/p}  (1.111)

where p ≥ 1, e.g., p = 1, 2, ∞. For convenience, we can drop p so that d(x, y) = ||x − y||.

• We first prove constructively the existence of a fixed point. As g(x) is a contraction, we have

||x_{n+1}−x_{n}|| = ||g(x_{n})−g(x_{n−1})|| ≤ k||x_{n}−x_{n−1}|| ≤ k^{2}||x_{n−1}−x_{n−2}|| ≤ ⋯ ≤ k^{n}||x_{1}−x_{0}||

(1.112)

As 0 ≤ k < 1, we have lim_{n→∞} ||x_{n+1} − x_{n}|| = 0. This is a Cauchy sequence (Section A.1) that converges to some point lim_{n→∞} x_{n} = x^{∗} also in the space. We further have

g(x^{∗}) = g( lim_{n→∞} x_{n}) = lim_{n→∞} g(x_{n}) = lim_{n→∞} x_{n+1} = x^{∗}  (1.113)

i.e., the limit of the Cauchy sequence lim_{n→∞} x_{n} = x^{∗} is a fixed point.

• We next prove the uniqueness of the fixed point. Let x_{1}^{∗} and x_{2}^{∗} be two fixed points of g(x), then we have

||g(x_{1}^{∗}) − g(x_{2}^{∗})|| ≤ k||x_{1}^{∗} − x_{2}^{∗}|| = k||g(x_{1}^{∗}) − g(x_{2}^{∗})||  (1.114)

For any k ≠ 0, the above holds only if ||x_{1}^{∗} − x_{2}^{∗}|| = ||g(x_{1}^{∗}) − f(x_{2}^{∗})|| = 0,

i.e., x_{1}^{∗} = x_{2}^{∗} is the unique fixed point.

QED Proof of the Contraction Mapping theorem: Consider the Taylor expansion (Section A.7) of the function g(x) in the neigh-

borhood of x^{∗}:

g(x) = g(x^{∗})+g′(x^{∗})(x−x^{∗})+R(x−x^{∗}) = g(x^{∗})+J_{g}(x^{∗})(x−x^{∗})+R(x−x^{∗})

(1.115)

where R(x − x^{∗}) is the remainder composed of second and higher order terms of δ = x − x^{∗}. Subtracting g(x^{∗}) and taking any p-norm on both sides, we get

||g(x) − g(x^{∗})||_{p} = ||J_{g}(x^{∗})(x − x^{∗}) + R(x − x^{∗})||_{p}  (1.116)

When x → x^{∗}, the second and higher order terms of x − x^{∗} disappear and R(x − x^{∗}) → 0, we have

||g(x) − g(x^{∗})||_{p} = ||J_{g}(x^{∗})(x − x^{∗})||_{p} ≤ ||J_{g}(x^{∗})||_{p} · ||x − x^{∗}||_{p}, (1.117)

The inequality is due to the Cauchy-Schwarz inequality (Section A.1) if ||J_{g}(x^{∗})|| < 1, the function g(x) is a contraction at x^{∗}.

QED

2 Unconstrained Optimization

Optimization, also known as mathematical programming, is a field of mathematical study with the general goal of obtaining the best solution of a given problem, and it finds a wide range of applications in many different areas, such as to minimize the error or cost, or to maximize the yields or profits. Mathematically an optimization problem can be formulated as to find the values of a set of variables represented by a vector x = [x_{1}, ⋯, x_{N}]^{T} so that an the objective function f(x) = f(x_{1}, ⋯, x_{N}) is either maximized or minimized. The solution of this problem, called the optimal solution, is denoted by x^{∗}. The fact that the function is either maximized or minimized is expressed as

x^{∗} = arg max_{x} f((x) i.e.  f(x^{∗}) ≥ f(x)  (2.1)

or

x^{∗} = arg min_{x} f((x) i.e.  f(x^{∗}) ≤ f(x)  (2.2)

The optimization problem is unconstrained if the variable x can take any value in the domain of the function, the entire domain ℝ^{N} of the objective function f(x), otherwise it is constrained if x is constrained to take values only in a subset of the domain, such as ||x|| < 5.

Optimization plays an essential role in machine learning as many learning algorithms can be formulated as solving an optimization problem so that certain objective or criterion function can be either minimized or maximized. In general, we desire to minimize the difference between the observed data and the model prediction by the learning method measured in some way (e.g. the sum of squared error), or to maximizae the similarity between them measured in some way (e.g., likelihood of the model parameters). This issue will be discussed first in Section 4.1, and then through out the book chapters wherever optimization is involved in the specific algorithms.

We will consider unconstrained optimization in this chapter and constrained optimization problems in the next chapter. We will discuss various methods used for solving optimization problems in the most abstract form of maximizing or minimizing a given objective function f(x), which will take some specific meanings in the context of various learning algorithms in the future chapters.

Two main types of algorithms widely used in unconstrained optimization are the gradient descent method and Newton’s method, both of which find the optimal solution iteratively based on certain approximation of the objective function. Specifically, the gradient descent method approximates the objective function by the first two terms for the constant and linear components of its Taylor expansion, while Newton’s method approximates the function by the first three terms of its Taylor expandion including the third term for the quadratic component as well (see Section A.7). In other words, the gradient method depends only on the gradient vector of the function representing the local slope of the corresponding hypersurface, while Newton’s method depends on the second order Hessian matrix describing more global shape of the surface as well as the local slope. Consequently, as Newton’s method approximates the function more accurately, it is in general more efficient as its iteration converges more quickly than the gradient method it requires the second order derivatives which may not be availabe as conveniently as the first order derivatives. In practice, either method can be used depending on tradeoff between algorithmic efficiency and the additional cost for obtaining some extra information.

2.1 Optimization by Solving Equations

Before discussing specific methods for optimization, we first show that an unconstrained minimization problem can be converted into a problem of equation solving and visa versa, i.e., these two types of problems are equivalent in this sense.

• To minimize an objective function f(x), we can set its gradient vector g_{f}(x) to zero and solve the resulting equation system:

g_{f}(x) = ▽_{x}f(x) = \frac{df(x)}{dx} = [ ∂f/∂x_{1} ; ⋮ ; ∂f/∂x_{N} ] = 0  (2.3)

The solution x^{∗} of this equation system, called a critical, stationary, or stable point of f(x), is also the solution to the optimization problem, if it is not a saddle point, that either maximizes or minimizes f(x).

This equation system can be solved by any of the methods discussed in the previous chapter, such as the Newton-Raphson method, which finds the root of a general function f(x) iteratively: x_{n+1} = x_{n} − J_{f}^{−1}(x_{n}) f(x_{n}). Here, specifically, to solve the equation g_{f}(x) = 0, we first get the Jacobian J_{g}(x) = H_{f}(x) of the gradient g_{f}(x) of f(x), which is the Hessian of f(x), and then carry out the iteration below to eventually find x^{∗} that minimizes f(x):

x_{n+1} = x_{n} − J_{g}^{−1}(x_{n}) g_{f}(x_{n}) = x_{n} − H_{f}^{−1}(x_{n}) g_{f}(x_{n})  (2.4)

The second equality is due to the fact that the Jacobian of the gradient g_{f} of function f(x) is the Hessian of the function, i.e., J_{g} = H_{f}.

• To solve an equation system f(x) = [f_{1}(x), ⋯, f_{N}(x)]^{T} = 0, we can minimize an objective function defined as

J(x) = \frac{1}{2}f^{T} (x)f(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2} ∑_{i=1}^{N} |f_{i}(x)|^{2}  (2.5)

The solution that minimizes J(x) is also the solution of the original equation. To see this, we again set the gradient of J(x) zero to get as above:

g_{J}(x) = ▽_{x}J(x) = \frac{d}{dx} [\frac{1}{2}f^{T} (x)f(x)] = f′(x) f(x) = J_{f} (x) f(x) = 0

(2.6)

As in general the Jacobian J_{f} (x) = f′(x) is a full rank matrix, the homogeneous equation above only has a zero solution f(x) = 0, which is indeed the original equation.

2.2 Newton’s Method

We first consider Newton’s method applied to minimizing a univariate function f(x). The Taylor series expansion (Section A.7) of the function f(x) around a point x_{0} is:

f(x) = f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2} + ⋯ + \frac{1}{n!}f^{(n)}(x_{0})(x − x_{0})^{n} + ⋯
≈ f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2} = q(x)  (2.7)

If f(x) is a quadratic function denoted by q(x), then its Taylor series contains only the first three terms (constant, linear, and quadratic terms). In this case, we can find the vertex point at which f(x) = q(x) reaches its extremum, by first setting its derivative to zero:

q′(x) = \frac{d}{dx}q(x) = \frac{d}{dx} [f(x_{0}) + f′(x_{0})(x − x_{0}) + \frac{1}{2}f′′(x_{0})(x − x_{0})^{2}]
= f′(x_{0}) + f′′(x_{0})(x − x_{0}) = 0  (2.8)

and then solving the resulting equation to get:

x^{∗} = x = x_{0} − \frac{f′(x_{0})}{f′′(x_{0})} = x_{0} + Δx_{0}  (2.9)

where Δx_{0} = −f′(x_{0})/f′′(x_{0}) is the step we need to take to move from any initial point x_{0} to the solution x^{∗} in a single step. Note that f(x^{∗}) is a minimum if f′′(x^{∗}) > 0 but a maximum if f′′(x^{∗}) < 0.

If f(x) ≠ q(x) is not quadratic, the result above can still be considered as an approximation of the solution, which can be improved iteratively from the initial guess x_{0} to eventually approach the solution:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − \frac{f′(x_{n})}{f′′(x_{n})}, n = 0, 1, 2, ⋯  (2.10)

Figure 2.1 Newton’s Method

where Δx_{n} = −f′(x_{n})/f′′(x_{n}) is the step size.

We see that in each iterative step, the function value f(x_{n}) is fitted by a quadratic functions q(x_{n}) and its vertex at x_{n+1} is used as the updated approximated solution, at which the function is again fitted by another quadratic function q(x_{n+1}) for the next iteration, as shown in Fig. 2.1. Through this process the solution can be approached as n → ∞.

We note that the iteration in Eq. (2.10) is the same as Eq. (1.47) in Chapter 1, if the function f(x) is replaced by its derivative f′(x). In other words, Newton’s method for optimization is actually solving the equation f′(x) = 0, the necessary condition for an extremum of f(x), by the Newton-Raphson method.

Newton’s method for the minimization of a single-variable function f(x) can be generalized for the minimization of a multivariable function f(x) = f(x_{1}, ⋯, x_{N}). Again, we approximate f(x) by a quadratic function q(x) containing only the first three terms of its Taylor series at some initial point x_{0}:

f(x) ≈ f(x_{0}) + g_{0}^{T} (x − x_{0}) + \frac{1}{2}(x − x_{0})^{T} H_{0} (x − x_{0}) = q(x)  (2.11)

where g_{0} and H_{0} are respectively the gradient vector and Hessian matrix of the function f(x) at x_{0}:

g_{0} = g_{f}(x_{0}) = \frac{d}{dx}f(x_{0}) = [ ⋮_{\frac{∂f(x_{0})}{∂x_{N}}}^{\frac{∂f(x_{0})}{∂x_{1}}} ] ,

H_{0} = H_{f}(x_{0}) = \frac{d}{dx}g(x_{0}) = \frac{d^{2}}{dx^{2}} f(x_{0}) = [ \begin{matrix}\frac{∂^{2}f(x_{0})}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x_{0})}{∂x_{1}∂x_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x_{0})}{∂x_{N} ∂x_{1}} & ⋯ & \frac{∂^{2}f(x_{0})}{∂x_{N}^{2}} \end{matrix} ](2.12)

The stationary point of f(x) = q(x) can be found from any initial guess x_{0} by setting its derivative to zero:

\frac{d}{dx}q(x) = \frac{d}{dx} [f(x_{0}) + g_{0}^{T} (x − x_{0}) + \frac{1}{2}(x − x_{0})^{T} H_{0} (x − x_{0})]
= g_{0} + H_{0} (x − x_{0}) = 0  (2.13)

and solving the resulting equation to get

x^{∗} = x = x_{0} − H(x_{0})^{−1}g(x_{0})  (2.14)

Similar to the single-variable case in which f(x^{∗}) is either a minimum or maximum depends on whether the second order derivative f′′(x^{∗}) is greater or smaller than zero, here in multi-variable case the function value f(x^{∗}) at the stationary point is a maximum, minimum, or saddle point, depending on the curvature of the function represented by the Hessian matrix H^{∗} = H_{f}(x)|_{x=x^{∗}} evaluated at x^{∗}:

• If H^{∗} > 0 is positive definite (all eigenvalues are positive), then f(x) is convex and f(x^{∗}) is the global minimum;

• If H^{∗} < 0 is negative definite (all eigenvalues are negative), then f(x is concave and f(x^{∗}) is the global maximum;

• If H^{∗} is indefinite (with both positive and negative eigenvalues), then f(x) is neither convex nor concave, and f(x^{∗}) is a saddle point (maximum in some directions, but minimum in others).

If f(x) is quadratic and convex, the optimization problem is called convex programming, and its minimum can be easily obtained. (Equivalently, if f(x) is quadratic and concave, its maximum can be easily obtained.)

But if f(x) ≠ q(x) is not quadratic, the result above is not a minimum or maximum, but it can still be used as an approximation of the solution, which can be improved iteratively to approach the solution:

x_{n+1} = x_{n} + Δx_{n} = x_{n} − H_{n}^{−1}g_{n} = x_{n} + d_{n}, n = 0, 1, 2, ⋯  (2.15)

where g_{n} = g(x_{n}), H_{n} = H(x_{n}), and the increment d_{n} = Δx_{n} = −H_{n}^{−1}g_{n} is called Newton search direction. We note that this iteration is just a generalization of x_{n+1} = x_{n} − f′(x_{n})/f′′(x_{n}) in 1-D case where g_{n} = f′(x_{n}) and H_{n} = f′′(x_{n}), and the iteration x_{n+1} = x_{n} − H_{n}^{−1} g_{n} above is just Eq. (1.68) in Chapter 1, the Newton-Raphson method applied to solving equation f′(x) = g_{f}(x) = 0, as the necessary condition for an extremum of f(x). The computational complexity for each iteration is O(N^{3}) due to the inverse operation H_{n}^{−1}.

For example, we can solve an over-constrained nonlinear equation system f(x) = 0 of M equations and N < M unknowns by minimizing the following SSE:

ε(x) = \frac{1}{2}||f(x)||^{2} = \frac{1}{2} ∑_{m=1}^{M} f_{m}^{2}(x)  (2.16)

To use the Newton-Raphson method, we first find the gradient of the error function:

g_{ε}(x) = \frac{d}{dx}ε(x) = \frac{1}{2} \frac{d}{dx}||f(x)||^{2} = \frac{df}{dx} f = J_{f}^{T} f  (2.17)

of which the ith component is:

g_{i} = \frac{∂}{∂x_{i}} (\frac{1}{2}||f||^{2}) = \frac{1}{2} ∑_{m=1}^{M} \frac{∂}{∂x_{i}} f_{m}^{2} = ∑_{m=1}^{M} \frac{∂f_{m}}{∂x_{i}} f_{m} = ∑_{m=1}^{M} J_{mi} f_{m} (2.18)

where J_{mi} = ∂f_{m}/∂x_{i} is the component of the function’s Jacobian J_{f}(x) in the mth row and ith column.

We further find the component of the Hessian H_{ε} of the error function in the ith row and jth column:

H_{ij} = \frac{∂^{2}}{∂x_{i}∂x_{j}} (\frac{1}{2}||f||^{2}) = \frac{∂}{∂x_{j}} g_{i} = ∑_{m=1}^{M} \frac{∂}{∂x_{j}} [f_{m} \frac{∂f_{m}}{∂x_{i}} ]

= ∑_{m=1}^{M} [\frac{∂f_{m}}{∂x_{i}} \frac{∂f_{m}}{∂x_{j}} + f_{m} \frac{∂^{2}f_{m}}{∂x_{i}∂x_{j}} ]

≈ ∑_{m=1}^{M} \frac{∂f_{m}}{∂x_{i}} \frac{∂f_{m}}{∂x_{j}} = ∑_{m=1}^{M} J_{mi}J_{mj}  (2.19)

Here J_{ij} is the component of the Jacobian J_{f} in the ith row and jth column, and we have dropped the second order terms f_{m} (∂^{2}f_{m}/∂x_{i}∂x_{j}). Now the Hessian matrix can be written as

H_{ε}(x) = \frac{d^{2}}{dx^{2}} ε(x) = \frac{d}{dx}g_{ε}(x) ≈ J_{f}^{T} J_{f}  (2.20)

Based on Newton-Raphson’s method, the optimal solution x^{∗} that minimizes ε(x) can be found iteratively:

x_{n+1} = x_{n} − H_{n}^{−1}g_{n} ≈ x_{n} − (J_{n}^{T} J_{n})^{−1}J_{n}^{T} f_{n}  (2.21)

This is the same as pseudo-inverse method in Eq. (1.77) in Chapter 1.

Example 2.1 Consider the following quadratic function for example:

q(x = q(x_{1}, x_{2}) = \frac{1}{2}x^{T} Ax = \frac{1}{2}[x_{1} x_{2}] [ \begin{matrix}a & b/2 \\ b/2 & c \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = \frac{1}{2}(ax_{1}^{2}+bx_{1}x_{2}+cx_{2}^{2})

The gradient vector and Hessian matrix of this function are respectively

g = [ \begin{matrix}ax_{1} + bx_{2}/2 \\ bx_{1}/2 + cx_{2} \end{matrix} ] , H = [ \begin{matrix}a & b/2 \\ b/2 & c \end{matrix} ] = A

The two eigenvalues of the symmetric matrix H can be found to be

λ_{1,2} = \frac{1}{2}[(a + c) ± \sqrt{(a + c)^{2} − 4ac + b^{2}}]  (2.22)

and the determinant of H is det H = λ_{1}λ_{2} = ac − b^{2}/4. The shape and extreme of the surface represented by q(x) are determined by the coefficients let a, b, c, which can take any values such as those discussed below and shown in Fig. 2.2.

• a = 3, b = 2, c = 4, λ_{1} = 2.382, λ_{2} = 4.618 are both positive, and H is positive definite. q(x) has a minimum at q(0, 0) = 0.

• a = −3, b = 2, c = −4, λ_{1} = −4.618 λ_{2} = −2.382 are both negative, and H is negative definite. q(x) has a maximum at q(0, 0) = 0.

• a = 3, b = \sqrt{4ac}, c = 4, λ_{1} = 7, λ_{2} = 0, and H is positive semi-definite. q(0, 0) is a mininum along one direction but constant along another.

• a = −3, b = 2, c = 4, λ_{1} = −3.140, λ_{2} = 4.140 are of different signs, and H is indefinite. q(0, 0) = 0 is a saddle point, which is a maximum along one direction but a minimum along another.

These properties based on the (semi) definiteness of the Hessian H can be generalized to quadratic functions with more variables than two (see Section A.2.3).

We can speed up the convergence by a bigger step size δ > 1. However, if δ is too big, the solution may be skipped and the iteration may not converge if it gets into an oscillation around the solution. Even worse, the iteration may become divergent. For such reasons, a smaller step size δ < 1 may be preferred sometimes.

In summary, Newton’s method approximates the function f(x) at an estimated solution x_{n} by a quadratic equation (the first three terms of the Taylor’s series) based on the gradient g_{n} and Hessian H of the function at x_{n}, and treat the vertex of the quadratic equation as the updated estimate x_{n+1}.

Newton’s method requires the Hessian matrix as well as the gradient to be available. Moreover, it is necessary calculate the inverse of the Hessian matrix in each iteration, which may be computationally expensive.

Example 2.2 The Newton’s method is applied to solving the following non-

Figure 2.2 Quadratic Surfaces The contour lines of the quadratic function are shown in color coded lines and the gradient vectors at the extreme/saddle points are shown in black.

linear equation system of N = 3 variables:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

with the exact solution (x_{1} = 0.5, x_{2} = 0, x_{3} = −0.5). These equations can be expressed in vector form as f(x) = 0 and solved as an optimization problem with the objective function J(x) = f^{T} (x)f(x). The iteration from an initial guess x_{0} = 0 is shown below.

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & ||f(x)|| \\ 0 & 0.000000, 0.000000, 0.000000 & 1.016120e + 01 \\ 1 & 0.500000, 0.500000, −0.500000 & 1.552502e + 02 \\ 2 & 0.499550, 0.250800, −0.493801 & 3.881300e + 01 \\ 3 & 0.500096, 0.126206, −0.496852 & 9.702208e + 00 \\ 4 & 0.500025, 0.063914, −0.498405 & 2.425198e + 00 \\ 5 & 0.500010, 0.032778, −0.499181 & 6.059054e − 01 \\ 6 & 0.500005, 0.017231, −0.499570 & 1.510777e − 01 \\ 7 & 0.500003, 0.009498, −0.499763 & 3.737330e − 02 \\ 8 & 0.500002, 0.005712, −0.499857 & 8.959365e − 03 \\ 9 & 0.500001, 0.003968, −0.499901 & 1.900145e − 03 \\ 10 & 0.500001, 0.003326, −0.499917 & 2.577603e − 04 \\ 11 & 0.500001, 0.003206, −0.499920 & 8.932714e − 06 \\ 12 & 0.500001, 0.003202, −0.499920 & 1.238536e − 08 \\ 13 & 0.500001, 0.003202, −0.499920 & 2.371437e − 14 \end{matrix}  (2.23)

We see that after 13 iterations the algorithm converges to the following approximated solution with accuracy of J(x) = ||f(x^{∗})||^{2} ≈ 10^{−28}:

x^{∗} = [ \begin{matrix}0.5000008539707297 \\ 0.0032017070323056 \\ −0.4999200212218281 \end{matrix} ]  (2.24)

2.3 Gradient Descent Method

Newton’s method discussed above is based on the gradient g_{f}(x) and Hessian H_{f}(x), representing the local variation and more global curvature of the function f(x) to be minimized. The method is not applicable if the Hessian H_{f} is not available, because either the analytical form of the function is not known or the cost for computing the inverse H_{f}^{−1} is too high. In such cases, we can use the gradient descent method based only on the gradient but no longer the Hessian.

We first consider the minimization of a single-variable function f(x). From any initial point x_{0}, we can move to a nearby point x along the opposite direction of the gradient, so that the function value will be smaller than that at the current position x_{0}:

x = x_{0} − δf′(x_{0}) = x_{0} + Δx_{0}, where Δx_{0} = −δf′(x_{0}), δ > 0 (2.25)

We see that no matter whether f′(x_{0}) is positive or negative, the function value f(x) approximated by the first two terms of its Taylor series is always reduced if the positive step size delta is small enough:

f(x) ≈ f(x_{0}) + f′(x_{0})Δx_{0} = f(x_{0}) − |f′(x_{0})|^{2}δ < f(x_{0})  (2.26)

This process can be carried out iteratively

x_{n+1} = x_{n} − δ_{n} f′(x_{n}),  n = 0, 1, 2, ⋯  (2.27)

until eventually reaching a point x^{∗} at which f′(x^{∗}) = 0 and no further progress can be made, i.e. a local minimum of the function is reached.

This simple method can be generalized to minimize a multi-variable objective function f(x) = f(x_{1}, ⋯, x_{N}) in N-D space. The derivative of the 1-D case is generalized to the gradient vector g(x) = df(x)/dx of function f(x), which is in the direction along which the function increases most rapidly with the steepest slope, perpendicular to the contour or iso-lines of the function f(x). The fastest way to reduce f(x) is to go down hill along the opposite direction of the gradient vector.

Specifically the gradient descent method (also called steepest descent or down hill method) carries out the iteration x = x_{0} + Δx, which is can be considered as the first two terms of the Taylor series of f(x) with Δx = −δg (δ > 0):

f(x) ≈ f(x_{0}) + g_{0}^{T} Δx = f(x_{0}) − δg_{0}^{T} g_{0} = f(x_{0}) − δ||g||^{2} < f(x_{0}) (2.28)

As the function value is always reduced by this iteration, it can be carried out repeatedly

x_{n+1} = x_{n} − δ_{n} g_{n} = (x_{n−1} − δ_{n−1} g_{n−1}) − δ_{n} g_{n} = ⋯ = x_{0} − ∑_{i=0}^{n} δ_{n} g_{i} (2.29)

until eventually reaching a point x^{∗} at which g(x^{∗}) = 0 and f(x) is minimized.

Comparing Eq. (2.29) with Newton’s method where the search direction is d_{n} = −H_{n}^{−1}g_{n} in Eq. (2.15) based on both H_{n} and g_{n}, we see that here in gradient descent method the search direction d_{n} = −g_{n} relies only on the local information contained in the gradient g without more global information contained in the second order Hessian H, i.e., the gradient method does not have as much information as Newton’s method and therefore may not be as efficient. For example, when the function is quadratic, as discussed before, Newton’s method can find the solution in a single step from any initial guess, but it takes the gradient descent method multiple steps to reach the solution, as it always follows the negative direction of the local gradient, which does not lead toward the solution directly in general. However, the gradient descent method is computationally less expensive as the Hessian matrix is not needed.

Example 2.3 Consider a two-variable quadratic function:

f(x) = x^{T} Ax = [x_{1} x_{2}] [ \begin{matrix}a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]
= [x_{1} x_{2}] [ \begin{matrix}2 & 1 \\ 1 & 1 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = 2x_{1}^{2} + 2x_{1}x_{2} + x_{2}^{2}

where Ais a symmetric positive definite matrix. This function has a minimum f(x_{1}, x_{2}) = 0 at x_{1} = x_{2} = 0.

Figure 2.3 Gradient of a Quadratic Surface

We select an initial guess x_{0} = [1, 2]^{T}, at which the gradient is g_{0} = [4, 3]^{T}, as shown in Fig. 2.3. Now we compare the gradient method with Newton’s method, in terms of the search direction d and progress of the iterations:

• Newton’s method: Here we have

g = [ \begin{matrix}2x_{1} + x_{2} \\ x_{1} + x_{2} \end{matrix} ] , H = A = [ \begin{matrix}2 & 1 \\ 1 & 1 \end{matrix} ] , H^{−1} = [ \begin{matrix}1 & −1 \\ −1 & 2 \end{matrix} ]

and the search direction is d_{0} = −H^{−1}g_{0} = −[1, 2]^{T} (the red arrow in the figure). In a single iteration we reach

x_{1} = x_{0} − H^{−1}g_{0} = [ \begin{matrix}1 \\ 2 \end{matrix} ] − [ \begin{matrix}1 & −1 \\ −1 & 2 \end{matrix} ] [ \begin{matrix}4 \\ 3 \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ]

which is the minimum of the function.

• The gradient descent method: the search direction is d_{0} = −g_{0} = −[4, 3]^{T}, perpendicular to the contour of the function. The first iteration is:

x_{1} = x_{0} − δg_{0} = [ \begin{matrix}1 \\ 2 \end{matrix} ] − δ [ \begin{matrix}4 \\ 3 \end{matrix} ] = [ \begin{matrix}1 − δ4 \\ 2 − δ3 \end{matrix} ]

We need to determine the step size δ (to be considered later) to find x_{1}, and then continue the iteration.

Fig. 2.4 compares the different search directions by Newton’s method (red) and the gradient descent method (blue). We see that the solution is reached in a single step in Newton’s method, based on the Hessian matrix H representing some global information of the elliptical shape of the contour line of the quadratic function, as well as the gradient g_{0} representing the local information at x_{0}. In comparison, the search direction of the gradient descent method is based on only the local information in the gradient g_{0} without more global information in H, consequently multiple steps are needed for the iterativon to approach the

Figure 2.4 Comparison of Gradient Descent and Newton’s Method

solution. The gradient descent method is less effective than Newton’s method due to the lack of second order derivative carrying more global information.

In the gradient descent method, we also need to determine a proper step size δ. If δ is too small, the iteration may converge too slowly, especially when f(x) reduces slowly toward its minimum. On the other hand, if δ is too large but f(x) has some rapid variations in the local region, the minimum of the function may be skipped and the iteration may not converge. We will consider how to find the optimal step size in the next section.

2.4 Line minimization

In general, in any iterative algorithm for minimizing a function f(x), including Newton’s method (Eq. (2.15)) and the gradient descent method (Eq. (2.29)), the variable x is updated iteratively x_{n+1} = x_{n} + δ_{n}d_{n} based on both the search direction d_{n} and the step size δ, which need to be determined in such a way that the function value f(x) is maximally reduced at each iteration.

First, the search direction d_{n} needs to point away from the gradient g_{n} along which f(x_{n}) increases most rapidly. In other words, the angle between d_{n} and g_{n} should be greater than π/2:

cos^{−1} (\frac{d_{n}^{T} g_{n}}{||d_{n}|| ||g_{n}||}) > \frac{π}{2}  i.e.,  d_{n}^{T} g_{n} < 0  (2.30)

This condition is indeed satisfied in both the gradient descent method with d_{n} = −g_{n} and Newton’s method with d_{n} = −H_{n}^{−1}g_{n}:

d_{n}^{T} g_{n} = −g_{n}^{T} H_{n}^{−1}g_{n} < 0,  d_{n}^{T} g_{n} = −g_{n}^{T} g_{n} = −||g_{n}||^{2} < 0  (2.31)

Here H_{n} is assumed to be positive definite so that a minimum exists.

Next, the optimal step size δ_{n} needs to be such that the function value f(x_{n+1}) = f(x_{n} + δ_{n}d_{n}) is minimized along the search direction d_{n}. To do so, we set to zero the derivative of the function value with respect to δ_{n}, and get the directional derivative (Section A.4.3) along the direction of d_{n} based on the chain rule:

\frac{d}{dδ_{n}} f(x_{n+1}) = \frac{d}{dδ_{n}} f(x_{n}+δ_{n}d_{n}) = (\frac{d f(x_{n+1})}{dx})^{T} \frac{d(x_{n} + δ_{n}d_{n})}{dδ_{n}} = g_{n+1}^{T}d_{n} = 0

(2.32)

This result indicates that ideally the gradient g_{n+1} = f′(x_{n+1}) at the next point x_{n+1} should be perpendicular to the search direction d_{n}. In other words, when traversing along d_{n}, we should stop at the point x_{n+1} = x_{n} + δd_{n} at which the gradient g_{n+1} is perpendicular to d_{n}, i.e., it has zero component along d_{n}, and the corresponding δ is the optimal step size.

To find such an optimal step size δ, we can consider f(x_{n+1}) = f(x_{n} +δd_{n}) as a function of δ treated as the single variable, and find the its optimal value that minimizes f(x_{n+1}). To do so, we assume the value of δ is small enough so that this function can be pproximated by the first three terms of its Taylor series at δ = 0 (Maclaurin series of function f(δ)):

f(x_{n+1}) = f(x_{n} + δd_{n})
≈ [f(x_{n} + δd_{n})]_{δ=0} + δ [\frac{d}{dδ} f(x_{n} + δd_{n})]_{δ=0} + \frac{δ^{2}}{2} [\frac{d^{2}}{dδ^{2}} f(x_{n} + δd_{n})]_{δ=0} (2.33)

The three terms of this expression can be further written as

[f(x_{n} + δd_{n})]_{δ=0} = f(x_{n})  (2.34)
[\frac{d}{dδ} f(x_{n} + δd_{n})]_{δ=0} = [g(x_{n} + δd_{n})^{T} d_{n}]_{δ=0} = g_{n}^{T} d_{n}  (2.35)
[\frac{d^{2}}{dδ^{2}} f(x_{n} + δd_{n})]_{δ=0} = [\frac{d}{dδ} g(x_{n} + δd_{n})^{T} ]_{δ=0} \frac{d}{dδ} (x_{n} + δd_{n})
= [\frac{d}{dx}g(x) \frac{d}{dδ} (x_{n} + δd_{n})]_{δ=0}^{T} d_{n}
= (H_{n}d_{n})^{T} d_{n} = d_{n}^{T} H_{n}d_{n}  (2.36)

where H_{n} = H_{n}^{T} is the Hessian matrix of f(x) at x_{n}. Substituting these back into Eq. (2.33) we get:

f(x_{n} + δd_{n}) ≈ f(x_{n}) + δ g_{n}^{T} d_{n} + \frac{δ^{2}}{2} d_{n}^{T} H_{n} d_{n}  (2.37)

To find the optimal δ that minimizes f(x_{n+1}) = f(x_{n} + δd_{n}), we first find its derivative with respect to δ and then set it to zero:

\frac{d}{dδ} f(x_{n} + δd_{n}) ≈ \frac{d}{dδ} (f(x_{n}) + δg_{n}^{T} d_{n} + \frac{δ^{2}}{2} d_{n}^{T} H_{n} d_{n})
= g_{n}^{T} d_{n} + δ d_{n}^{T} H_{n} d_{n} = 0  (2.38)

Solving this equation for δ we get:

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}}  (2.39)

Note that only if the value of this δ_{n} is small enough for the approximation in Eq. (2.33) to be good enough, will it be a good approximation of the true optimal step size (and each search directions is perpendicular to the previous one). Otherwise it may not be be very different from the true optimal step size.

Based on this result, we can get the optimal step side for both the Newton’s method and gradient descent method as shown below:

• Newton’s method:

The search direction is d_{n} = −H_{n}^{−1}g_{n}, and the optimal step size is

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}} = \frac{g_{n}^{T} (H_{n}^{−1}g_{n})}{(H_{n}^{−1}g_{n})^{T} H_{n}(H_{n}^{−1}g_{n})} = \frac{g_{n}^{T} (H_{n}^{−1}g_{n})}{(H_{n}^{−1}g_{n})^{T} g_{n}} = 1 (2.40)

This confirms that the iteration in Eq. (2.15) is indeed optimal:

x_{n+1} = x_{n} + δ_{n}d_{n} = x_{n} − δ_{n}H_{n}^{−1}g_{n} = x_{n} − H_{n}^{−1}g_{n}  (2.41)

• Gradient descent method:

The search direction is d_{n} = −g_{n}, and Eq. (2.32) becomes:

g_{n+1}^{T}d_{n} = −g_{n+1}^{T}g_{n} = 0, i.e., g_{n+1} ⊥ g_{n}  (2.42)

i.e., the search direction d_{n+1} = −g_{n+1} is always perpendicular to the previous one d_{n} = −g_{n}, i.e., the iteration follows a zigzag path composed of a sequence of segments from the initial guess to the final solution. The optimal step size is

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} H_{n}d_{n}} = \frac{g_{n}^{T} g_{n}}{g_{n}^{T} H_{n}g_{n}} = \frac{||g_{n}||^{2}}{g_{n}^{T} H_{n}g_{n}}  (2.43)

Given H_{n} as well as g_{n}, we can find the optimal step size δ_{n} with complexity O(N^{2}) for each iteration. However, as we assume the Hessian H_{n} is not available in the gradient descent method, the optimal step size above cannot be computed. We can instead approximate f′′(δ)|_{δ=0} in the third term of Eq. (2.33) at two nearby points at δ = 0 and δ = σ, where σ is a small value:

[\frac{d^{2}}{dδ^{2}} f(x + δd)]_{δ=0} = [\frac{d}{dδ} f′(x + δd)]_{δ=0} = lim_{σ→0} \frac{f′(x + σd) − f′(x)}{σ}
≈ \frac{g^{T} (x + σd) d − g^{T} (x) d}{σ} = \frac{g_{σ}^{T} d − g^{T} d}{σ}  (2.44)

where g = g(x) and g_{σ} = g(x + σd). This approximation can be used to replace d_{n}^{T} H_{n}d_{n} in Eq. (2.38) above:

\frac{d}{dδ} f(x_{n} + δd_{n}) ≈ g_{n}^{T} d_{n} + \frac{δ}{σ} (g_{σn}^{T}d_{n} − g_{n}^{T} d_{n}) = 0  (2.45)

Solving for δ we get the estimated optimal step size:

δ_{n} = −\frac{σ g_{n}^{T} d_{n}}{g_{σn}^{T}d_{n} − g_{n}^{T} d_{n}} = −\frac{σ g_{n}^{T} d_{n}}{(g_{σn} − g_{n})^{T} d_{n}}  (2.46)

Specifically in the gradient descent method with d_{n} = −g_{n}, this optimal step size becomes:

δ_{n} = −\frac{σ g_{n}^{T} d_{n}}{(g_{σn} − g_{n})^{T} d_{n}} = −\frac{σ g_{n}^{T} g_{n}}{(g_{σn} − g_{n})^{T} g_{n}} = \frac{σ ||g_{n}||^{2}}{||g_{n}||^{2} − g_{σn}^{T}g_{n}}  (2.47)

and the iteration becomes

x_{n+1} = x_{n} − δ_{n}g_{n} = x_{n} + \frac{σ ||g_{n}||^{2}}{g_{σn}^{T}g_{n} − ||g_{n}||^{2}} g_{n}  (2.48)

Example 2.4 The gradient descent method applied to solve the same threevariable equation system previously solved by Newton’s method:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

The step size δ_{n} is determined by the secant method with σ = 10^{−6}. The iteration from an initial guess x_{0} = 0 is shown below:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & ||f(x)|| \\ 0 & 0.0000, 0.0000, 0.0000 & 1.032500e + 02 \\ 10 & 0.4246, −0.0073, −0.5002 & 1.535939e − 01 \\ 20 & 0.5015, 0.0064, −0.4998 & 2.448241e − 05 \\ 30 & 0.5009, 0.0057, −0.4998 & 9.178209e − 06 \\ 40 & 0.5006, 0.0052, −0.4998 & 4.17587e − 06 \\ 50 & 0.5004, 0.0049, −0.4999 & 2.122594e − 06 \\ 60 & 0.5003, 0.0047, −0.4999 & 1.182466e − 06 \\ 100 & 0.5001, 0.0043, −0.4999 & 1.805871e − 07 \\ 150 & 0.5000, 0.0041, −0.4999 & 2.720744e − 08 \\ 200 & 0.5000, 0.0041, −0.4999 & 4.934027e − 09 \\ 250 & 0.5000, 0.0040, −0.4999 & 9.630085e − 10 \\ 300 & 0.5000, 0.0040, −0.4999 & 1.939747e − 10 \\ 350 & 0.5000, 0.0040, −0.4999 & 3.961646e − 11 \\ 400 & 0.5000, 0.0040, −0.4999 & 8.140393e − 12 \\ 450 & 0.5000, 0.0040, −0.4999 & 1.677968e − 12 \\ 500 & 0.5000, 0.0040, −0.4999 & 3.462695e − 13 \\ 550 & 0.5000, 0.0040, −0.4999 & 7.136628e − 14 \\ 600 & 0.5000, 0.0040, −0.4999 & 1.474205e − 14 \end{matrix}

We see that after the first 100 iterations the error is reduced to about J(x) = ||f(x^{∗})||^{2} ≈ 10^{−14}, and With 500 additional iterations the algorithm converges to the following approximated solution with accuracy of J(x) ≈ 10^{−28}:

x^{∗} = [ \begin{matrix}0.5000013623816102 \\ 0.0040027495837189 \\ −0.4999000311539049 \end{matrix} ]  (2.49)

Although the gradient descent method requires many more iterations than Newton’s method to converge, the computational cost in each iteration is much reduced, as no more matrix inversion is needed.

When it is difficult or too computationally costly to find the optimal step size along the search direction, some suboptimal step size may be acceptable, such as in the quasi-Newton methods (Section 2.5) for minimization problems. In this case, although the step size δ is no longer required to be such that the function value at the new position f(x_{n+1}) = f(x_{n} + δd_{n}) is minimized along the search direction d_{n}, the step size δ still has to satisfy the following Wolfe conditions:

• Sufficient decrease (Armijo rule):

f(x_{n+1}) = f(x_{n} + δd_{n}) ≤ f(x_{n}) + c_{1} δd^{T} g_{n}  (2.50)

• Curvature condition:

g_{n+1}^{T}d_{n} ≥ c_{2} g_{n}^{T} d_{n}  (2.51)

As g_{n}^{T} d_{n} < 0, this condition can also be written in the following alternative form:

|g_{n+1}^{T}d_{n}| < |c_{2} g_{n}^{T} d_{n}|  (2.52)

Here the two constants c_{1} and c_{2} above satisfy 0 < c_{1} < c_{2} < 1.

In general, these conditions are motivated by the desired effect that after each iterative step, the function should have a shallower slope along d_{n}, as well as a lower value, so that eventually the solution can be approached where f(x) is minimum and the gradient is zero.

Specifically, to understand the first condition above, we represent the function to be minimized as a single-variable function of the step size ϕ(δ) = f(x_{n} +δd_{n}), and its tangent line at the point δ = 0 as a linear function L_{0}(δ) = a + bδ, where the intercept a can be found as a = L_{0}(0) = ϕ(δ)|_{δ=0} = f(x_{n}), and the slope b can be found as the derivative of ϕ(δ) = f(x_{n} + δd_{n}) at δ = 0:

b = \frac{d}{dδ} f(x_{n} + δd_{n})|_{δ=0} = g_{n}^{T} d_{n} < 0  (2.53)

which is required to be negative for the function value to be reduced, f(x_{n} + δd_{n}) < f(x_{n}). Now the function of the tangent line can be written as

L_{0}(δ) = a + bδ = f(x_{n}) + g_{n}^{T} d_{n}δ  (2.54)

Comparing this with a constant line L_{1}(δ) = f(x_{n}) of slope zero, we see that any straight line between L_{0}(δ) and L_{1}(δ) can be described by L(δ) = f(x_{n}) +

Figure 2.5 Iterations of the Gradient Descent Method

c_{1} g_{n}^{T} d_{n}δ with 0 < c_{1} < 1, with a slope 0 < c_{1}g_{n}^{T} d_{n} < g_{n}^{T} d_{n}. The Armijo rule is to find any δ > 0 that satisfies

f(x_{n+1}) = f(x_{n} + δd_{n}) ≤ f(x_{n}) + c_{1} g_{n}^{T} d_{n}δ < f(x_{n})  (2.55)

We see that the value of f(x) is guaranteed to be reduced.

The second condition requires that at the new position x_{n+1} the slope of the gradient g_{n+1} along the search direction d_{n} be sufficiently reduced to be less than a specified value (determined by c_{2}), in comparison to the slope at the old position x_{n}.

The reason why g_{n+1} ⊥ g_{n} can also be explained geometrically. As shown in Fig. 2.5, the gradient vectors at various points along the direction of −g_{n} are shown by the blue arrows, and their projections onto the direction represent the slopes of the function f(δ_{n}) = f(x_{n} − δ_{n}g_{n}). Obviously at x_{n+1} where f(δ_{n}) reaches its minimum, its slope is zero. In other words, the projection of the gradient g_{n+1} onto the direction of −g_{n} is zero, i.e., g_{n+1} ⊥ g_{n} or g_{n+1}^{T}g_{n} = 0.

The gradient descent method gradually approaches a solution x of an N-D minimization problem by moving from the initial guess x_{0} along a zigzag path composed of a set of segments with any two consecutive segments perpendicular to each other. The number of steps depends greatly on the initial guess. As illustrated in Fig. 2.6 for a quadratic surface over a 2-D space, the best possible case (blue) is that the solution happens to be on the gradient direction of the initial guess, which could be reached in a single step, while the worst possible case (red) is that the gradient direction of the initial guess happens to be 45 degrees off from the gradient direction of the optimal case, and it takes many

Figure 2.6 Best (blue) and Worst (red) Initial Guess

zigzag steps to go around the optimal path to reach the solution. Many of the steps are in the same direction as some of the previous steps.

To improve the performance of the gradient descent method we can include in the iteration a momentum term representing the search direction previously traversed:

x_{n+1} = x_{n} − δ_{n}g_{n} + α_{n}(x_{n} − x_{n−1})  (2.56)

Now two consecutive search directions are no longer perpendicular to each other and the resulting search path is smoother than the zigzag path without the momentum term. The parameter α_{m} controls how much momentum is to be added.

Obviously it is most desirable not to repeat any of the previous directions traveled so that the solution can be reached in N steps, each in a unique direction in the N-D space. In other words, the subsequent steps are independent of each other, never interfering with the results achieved in the previous steps. Such a method will be discussed in the next section.

Example 2.5 The Rosenbrock function

f(x_{1}, x_{2}) = (a − x_{1})^{2} + b(x_{2} − x_{1}^{2})^{2} (e.g., a = 1, b = 100)

is a two-variable non-convex function with a global minimum f(x_{1}, x_{2}) = 0 at the point (1, 1), which is inside a long parabolic shaped valley as shown in the left panel of Fig. 2.7. As the slope along the valley is very shallow, it is difficult for an algorithm to converge quickly to the minimum. For this reason, the Rosenbrock function is often used to test various minimization algorithms.

Bothe the gradient method and Newton’s method are used to find the min-

Figure 2.7 Rosenbrock Function (Left) and the Search Paths to Its Minimum (Right) The search paths for Newton’s and gradient descent methods are shown in red and blue respectively.

imum from an initial guess at (−0.5, 1.5). While it takes only a few iteration steps for the Newton’s method to converge to the minimum, following the search path shown in red in the right panel of Fig. 2.7, it takes many more steps for the gradient descent method to converge to the minimum, following the search path shown in blue in the right panel. Also, Fig. 2.8 shows a zoomed-in version of the last stage of the search path toward the final solution. We see that when the step size δ is small, the zigzag search path is composed of a long sequence consecutive segments perpendicular to each other.

2.5 Quasi-Newton Methods

As we have seen above, Newton’s methods can be used to solve both nonlinear systems to find roots of a set of simultaneous equations, and optimization problems to minimize a scalar-valued objective function based on the iterations of the same form. Specifically,

Figure 2.8 Search Path of the Gradient Descent (zoomed-in)

• Solving equations f(x) = 0:

x_{n+1} = x_{n} − J(x_{n})^{−1}f(x_{n}) = x_{n} − J_{n}^{−1}f_{n}  (2.57)

where J_{n} = J(x_{n}) is the Jacobian matrix of f(x_{n}),

• Minimizing f(x):

x_{n+1} = x_{n} − H(x_{n})^{−1}g(x_{n}) = x_{n} − H_{n}^{−1}g_{n}  (2.58)

where g_{n} = g(x_{n}) and H_{n} = H(x_{n}) are the gradient vector and Hessian matrix of f(x) at x_{n}, respectively.

If the Jacobian J(x) or the Hessian matrix H(x) is not available, or if it is too computationally costly to calculate its inverse (with complexity O(N^{3})), the quasi-Newton methods can be used to approximate the Hessian matrix or its inverse based only on the first order derivative, the gradient g of f(x) (with complexity O(N^{2})), similar to Broyden’s method previously considered (Section 1.5).

In the following, we consider the minimization of a function f(x). Its Taylor expansion around point x_{n+1} is

f(x) = f(x_{n+1})+(x−x_{n+1})^{T} g_{n+1}+\frac{1}{2}(x−x_{n+1})^{T} H_{n+1}(x−x_{n+1})+O(||x−x_{n+1}||^{3})

(2.59)

Taking derivative with respect to x, we get

\frac{d}{dx}f(x) = g(x) = g_{n+1} + H_{n+1}(x − x_{n+1}) + O(||x − x_{n+1}||^{2})  (2.60)

Evaluating at x = x_{n}, we have g(x_{n}) = g_{n}, and the above can be written as

g_{n+1} − g_{n} = H_{n+1}(x_{n+1} − x_{n}) + O(||x_{n+1} − x_{n}||^{2})
= B_{n+1}(x_{n+1} − x_{n})  (2.61)

where matrix B_{n} is the secant approximation of the Hessian matrix H_{n}, and the last equality is called the secant equation. For convenience, we further define:

s_{n} = x_{n+1} − x_{n}, y_{n} = g_{n+1} − g_{n}  (2.62)

so that the equation above can be written as

B_{n+1}s_{n} = y_{n}, or B_{n+1}^{−1}y_{n} = s_{n}  (2.63)

This is the quasi-Newton equation, which is the secant condition that must be satisfied by matrix B_{n}, or its inverse B_{n}^{−1}, in any of the quasi-Newton algorithms, all taking the follow general steps:

1. Initialize x_{0} and B_{0}, set n = 0;

2. Compute gradient g_{n} and the search direction d_{n} = −B_{n}^{−1}g_{n};

3. Get x_{n+1} = x_{n} + δd_{n} with step size δ satisfying the Wolfe conditions

4. update B_{n+1} = B_{n} + ΔB_{n} or B_{n+1}^{−1} = B_{n}^{−1} + ΔB_{n}^{−1}, that satisfies the quasi-Newton equation;

5. If termination condition is not satisfied, n = n + 1, go back to second step.

For this iteration to converge to a local minimum of f(x), B_{n} must be positive definite matrix, same as the Hessian matrix H it approximates. Specially, if B_{n} = I, then d_{n} = −g_{n}, the algorithm becomes the gradient descent method; also, if B_{n} = H_{n} is the Hessian matrix, then the algorithm becomes the Newton’s method.

In a quasi-Newton method, we can choose to update either B_{n} or its inverse B_{n}^{−1} based on one of the two forms of the quasi-Newton equation. We note that there is a dual relationship between the two ways to update, i.e., by swapping y_{n} and s_{m}, an update formula for B_{n} can be directly applied to its inverse and vice versa.

• The Symmetric Rank 1 (SR1) Algorithm Here matrix B_{n} is updated by an additional term:

B_{n+1} = B_{n} + uu^{T}  (2.64)

where u a vector and uu^{T} is a matrix of rank 1. Imposing the secant condition, we get

B_{n+1}s_{n} = B_{n}s_{n} + uu^{T} s_{n} = y_{n}, i.e., u(u^{T} s_{n}) = y_{n} − B_{n}s_{n} (2.65)

This equation indicates u is in the same direction as w = y_{n} − B_{n}s_{n}, and can therefore be written as u = cw. Substituting this back we get

uu^{T} s_{n} = c^{2}ww^{T} s_{n} = w  (2.66)

Solving this we get c = 1/(w^{T} s_{n})^{1/2},

u = cw = \frac{y_{n} − B_{n}s_{n}}{(w^{T} s_{n})^{1/2}} = \frac{y_{n} − B_{n}s_{n}}{((y_{n} − B_{n}s_{n})^{T} s_{n})^{1/2}}  (2.67)

and the iteration of B_{n} becomes

B_{n+1} = B_{n} + uu^{T} = B_{n} + \frac{(y_{n} − B_{n}s_{n})(y_{n} − B_{n}s_{n})^{T}}{(y_{n} − B_{n}s_{n})^{T} s_{n}}  (2.68)

Applying the Sherman-Morrison formula (Section A.2.4) we further get the inverse B_{n+1}:

B_{n+1}^{−1} = (B_{n} + uu^{T} )^{−1} = B_{n}^{−1} − \frac{B_{n}^{−1}uu^{T} B_{n}^{−1}}{1 + u^{T} B_{n}^{−1}u}
= B_{n}^{−1} + \frac{(s_{n} − B_{n}^{−1}y_{n})(s_{n} − B_{n}^{−1}y_{n})^{T}}{(s_{n} − B_{n}^{−1}y_{n})^{T} y_{n}}  (2.69)

We note that this equation is dual to the previous one, and it can also be obtained based on the duality between the secant conditions for B_{n} and B_{n}^{−1}. Specifically, same as above, we impose the secant condition for the inverse B_{n+1}^{−1} of the following form

B_{n+1}^{−1} = B_{n}^{−1} + uu^{T}  (2.70)

and get

B_{n+1}^{−1}y_{n} = B_{n}^{−1}y_{n} + uu^{T} y_{n} = s_{n},  (2.71)

Then by repeating the same process above, we get the same update formula for B_{n}^{−1}.

This is the formula for directly updating B_{n}^{−1}. For the iteration to converge to a local minimum, matrix B_{n+1}^{−1} needs to be positive definite as well as B_{n}^{−1}, we therefore require

w^{T} y_{n} = (s_{n} − B_{n}^{−1}y_{n})^{T} y_{n} = s_{n}^{T} y_{n} − y_{n}^{T} B_{n}^{−1}y_{n} > 0  (2.72)

It may be difficult to maintain this requirement through out the iteration. This problem can be avoided in the following DFP and BFGS methods.

• The BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm

This is a rank 2 algorithm in which matrix B_{n} is updated by two rank-1 terms:

B_{n+1} = B_{n} + αuu^{T} + βvv^{T}  (2.73)

Imposing the secant condition for B_{n}, we get

B_{n+1}s_{n} = (B_{n} + αuu^{T} + βvv^{T})s_{n} = B_{n}s_{n} + u(αu^{T} s_{n}) + v(βv^{T} s_{n}) = y_{n}

(2.74)

or

u(αu^{T} s_{n}) + v(βv^{T} s_{n}) = y_{n} − B_{n}s_{n}  (2.75)

which can be satisfied if we let

u = y_{n}, α = \frac{1}{s_{n}^{T} y_{n}} , v = B_{n}s_{n}, β = −\frac{1}{v^{T} s_{n}} = −\frac{1}{s_{n}^{T} B_{n}s_{n}} (2.76)

Substituting these into B_{n+1} = B_{n} + αuu^{T} + βvv^{T} we get

B_{n+1} = B_{n} + αuu^{T} + βvv^{T} = B_{n} + \frac{y_{n}y_{n}^{T}}{y_{n}^{T} s_{n}} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}}  (2.77)

Given this update formula for B_{n}, we can further find its inverse B_{n}^{−1} and then the search direction d_{n} = −B_{n}^{−1}g_{n}.

Alternatively, given B_{n}, we can further derive an update formula directly for the inverse matrix B_{n}^{−1}, so that the search direction d_{n} can be obtained without carrying out the inversion computation. Specifically, we first define U = [u_{1} u_{2}] and V = [v_{1} v_{2}], where

u_{1} = v_{1} = \frac{y_{n}}{(s_{n}^{T} y_{n})^{1/2}} , u_{2} = −v_{2} = \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}}  (2.78)

so that the expression above can be written as:

B_{n+1} = B_{n} + u_{1}v_{1}^{T} + u_{2}v_{2}^{T} = (B_{n} + UV^{T})^{−1}  (2.79)

and then apply the Sherman-Morrison formula to get:

B_{n+1}^{−1} = (B_{n} + UV^{T})^{−1} = B_{n}^{−1} − B_{n}^{−1}U(I + V^{T} B_{n}^{−1}U)^{−1}V^{T} B_{n}^{−1}
= B_{n}^{−1} − B_{n}^{−1}UC^{−1}V^{T} B_{n}^{−1}
= B_{n}^{−1} − B_{n}^{−1}[u_{1} u_{2}]C^{−1}(B_{n}^{−1}[v_{1} v_{2}])^{T}  (2.80)

where we have defined

C = [ \begin{matrix}c_{11} & c_{12} \\ c_{21} & c_{22} \end{matrix} ] = I + V^{T} B_{n}^{−1}U = I + [v_{1} v_{2}]^{T} B_{n}^{−1}[u_{1} u_{2}] (2.81)

with

c_{11} = 1 + v_{1}^{T} B_{n}^{−1}u_{1} = 1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}}  (2.82)

c_{22} = 1 + v_{2}B_{n}^{−1}u_{2} = 1 − \frac{s_{n}^{T} B_{n}B_{n}^{−1}B_{n}s_{n}}{s_{n}^{T} B_{n}s_{n}} = 0  (2.83)

c_{12} = v_{1}^{T} B_{n}^{−1}u_{2} = \frac{y_{n}^{T} B_{n}^{−1}B_{n}s_{n}}{(s_{n}^{T} y_{n})^{1/2}(s_{n}^{T} B_{n}s_{n})^{1/2}} = \frac{(s_{n}^{T} y_{n})^{1/2}}{(s_{n}^{T} B_{n}s_{n})^{1/2}}  (2.84)
c_{21} = v_{2}^{T} B_{n}^{−1}u_{1} = −c_{12}  (2.85)
C = [ \begin{matrix}c_{11} & c_{12} \\ −c_{12} & 0 \end{matrix} ] ,  C^{−1} = [ \begin{matrix}0 & −1/c_{12} \\ 1/c_{12} & c_{11}/c_{12}^{2} \end{matrix} ]  (2.86)

Substituting this C^{−1} into the expression for B_{n+1}^{−1}, we get:

B_{n+1}^{−1} = B_{n}^{−1} − [B_{n}^{−1}u_{1} B_{n}^{−1}u_{2}] [ \begin{matrix}0 & −1/c_{12} \\ 1/c_{12} & c_{11}/c_{12}^{2} \end{matrix} ] [B_{n}^{−1}v_{1} B_{n}^{−1}v_{2}]^{T}
= B_{n}^{−1} − \frac{1}{c_{12}} [B_{n}^{−1}u_{2}v_{1}^{T} B_{n}^{−1} − B_{n}^{−1}u_{1}v_{2}^{T} B_{n}^{−1}] − \frac{c_{11}}{c_{12}^{2}} B_{n}^{−1}u_{2}v_{2}^{T} B_{n}^{−1} (2.87)

where

\frac{1}{c_{12}} B_{n}^{−1}u_{2}v_{1}^{T} B_{n}^{−1} = \frac{(s_{n}^{T} B_{n}s_{n})^{1/2}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} \frac{y_{n}^{T}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} = \frac{s_{n}y_{n}^{T} B_{n}^{−1}}{s_{n}^{T} y_{n}}

(2.88)

−\frac{1}{c_{12}} B_{n}^{−1}u_{1}v_{2}^{T} B_{n}^{−1} = \frac{(s_{n}^{T} B_{n}s_{n})^{1/2}}{(s_{n}^{T} y_{n})^{1/2}} B_{n}^{−1} \frac{y_{n}}{(s_{n}^{T} y_{n})^{1/2}} \frac{s_{n}^{T} B_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} B_{n}^{−1} = \frac{B_{n}^{−1}y_{n}s_{n}^{T}}{s_{n}^{T} y_{n}}

(2.89)

\frac{c_{11}}{c_{12}^{2}} B_{n}^{−1}u_{2}v_{2}^{T} B_{n}^{−1} = − (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}^{T} B_{n}s_{n}}{s_{n}^{T} y_{n}} B_{n}^{−1} \frac{B_{n}s_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} \frac{s_{n}^{T} B_{n}}{(s_{n}^{T} B_{n}s_{n})^{1/2}} B_{n}^{−1}
= − (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}}  (2.90)

Substituting these three terms back into Eq. (2.87), we get the update formula for B_{n}^{−1}:

B_{n+1}^{−1} = B_{n}^{−1} − \frac{B_{n}^{−1}y_{n}s_{n}^{T} + s_{n}y_{n}^{T} B_{n}^{−1}}{s_{n}^{T} y_{n}} + (1 + \frac{y_{n}^{T} B_{n}^{−1}y_{n}}{s_{n}^{T} y_{n}} ) \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} (2.91)

In summary, B_{n+1}^{−1} can be found by either Eq. (2.77) for B_{n+1} followed by an additional inversion operation, or Eq. (2.91) directly for B_{n+1}^{−1}. The results obtained by these methods are equivalent.

• The DFP (Davidon-Fletcher-Powell) Algorithm

Same as the BFGS algorithm, the DFP algorithm is also a rank 2 algorithm, where instead of B_{n}, the inverse matrix B_{n}^{−1} is updated by two rank-1 terms:

B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T}  (2.92)

where α and β are real scalars, u and v are vectors. Imposing the secant condition for the inverse of B_{n}, we get

B_{n+1}^{−1}y_{n} = (B_{n}^{−1}+αuu^{T} +βvv^{T})y_{n} = B_{n}^{−1}y_{n}+u(αu^{T} y_{n})+v(βv^{T} y_{n}) = s_{n}

(2.93)

or

u(αu^{T} y_{n}) + v(βv^{T} y_{n}) = s_{n} − B_{n}^{−1}y_{n}  (2.94)

which can be satisfied if we let

u = s_{n}, α = \frac{1}{s^{T} y_{n}} , v = B_{n}^{−1}y_{n}, β = −\frac{1}{v^{T} y_{n}} = −\frac{1}{y_{n}^{T} B_{n}^{−1}y_{n}}

(2.95)

Substituting these into B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T} we get

B_{n+1}^{−1} = B_{n}^{−1} + αuu^{T} + βvv^{T} = B_{n}^{−1} + \frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} − \frac{B_{n}^{−1}y_{n}y_{n}^{T} B_{n}^{−1}}{y_{n}^{T} B_{n}^{−1}y_{n}} (2.96)

Following the same procedure used for the BFGS method, we can also obtain the update formula of matrix B_{n} for the DFP method:

B_{n+1} = B_{n} − \frac{B_{n}s_{n}y_{n}^{T} + y_{n}s_{n}^{T} B_{n}}{y_{n}^{T} s_{n}} + (1 + \frac{s_{n}^{T} B_{n}s_{n}}{y_{n}^{T} s_{n}} ) \frac{y_{n}y_{n}^{T}}{y_{n}^{T} s_{n}}  (2.97)

We note that Eqs. (2.96) and (2.77) form a duality pair, and Eqs. (2.97) and (2.91) form another duality pair. In other words, the BFGS and FDP methods are dual of each other.

In both the DFP and BFGS methods, matrix B_{n}^{−1}, as well as B_{n}, must be positive definite, i.e., z^{T} B_{n}z > 0 must hold for any z ≠ 0. We now prove that this requirement is satisfied if the curvature condition of the Wolfe conditions is satisfied:

g_{n+1}^{T}d_{n} ≥ c_{2} g_{n}^{T} d_{n} i.e. (g_{n+1} − c_{2} g_{n})^{T} d_{n} ≥ 0  (2.98)

Replacing the search direction d_{n} by δd_{n} = x_{n+1} − x_{n} = s_{n}, we get:

(g_{n+1} − c_{2}g_{n})^{T} s_{n} = g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n} ≥ 0  (2.99)

We also have

y_{n}^{T} s_{n} = (g_{n+1} − g_{n})^{T} s_{n} = g_{n+1}^{T}s_{n} − g_{n}^{T} s_{n}  (2.100)

Subtracting the first equation from the second, we get

y_{n}^{T} s_{n} − (g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n}) = (c_{2} − 1)g_{n}^{T} s_{n} > 0  (2.101)

The last inequality is due to the fact thatg_{n}^{T} s_{n} < 0 and c_{2} < 1. We therefore have

y_{n}^{T} s_{n} ≥ g_{n+1}^{T}s_{n} − c_{2}g_{n}^{T} s_{n} ≥ 0  (2.102)

Given y_{n}^{T} s_{n} > 0, we can further prove by induction that both B_{n+1} and B_{n+1}^{−1} based respectively on the update formulae in Eqs. (2.77) and (2.96) are positive definite. We first assume B_{0} and B_{n} are both positive definite, and express B_{n} in terms of its Cholesky decomposition, B_{n} = LL^{T}. We further define a = L^{T} z, b = L^{T} s, and get

a^{T} a = z^{T} B_{n}z, b^{T} b = s^{T} B_{n}s, a^{T} b = z^{T} B_{n}s  (2.103)

Now we can show that the sum of the first and third terms of Eq. (2.91) is positive definite:

z^{T} [B_{n} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}} ] z = z^{T} B_{n}z − \frac{(z^{T} B_{n}s_{n})^{2}}{s_{n}^{T} B_{n}s_{n}} = a^{T} a − \frac{(a^{T} b)^{2}}{b^{T} b} ≥ 0 (2.104)

The last step is due to the Cauchy-Schwarz inequality. Also, as s_{n}^{T} y_{n} ≥ 0, the second term of Eq. (2.77) is positive definite

z^{T} (\frac{s_{n}s_{n}^{T}}{s_{n}^{T} y_{n}} ) z ≥ 0  (2.105)

Figure 2.9 Search Path of SR1 based on B^{−1}

Combining these two results, we get

z^{T} B_{n+1}z = z^{T} (B_{n} − \frac{B_{n}s_{n}s_{n}^{T} B_{n}}{s_{n}^{T} B_{n}s_{n}} ) z + z^{T} (\frac{y_{n}y_{n}^{T}}{s_{n}^{T} y_{n}} ) z ≥ 0  (2.106)

i.e., B_{n+1} based on the update formula Eq. (2.77) is positive definite. Following the same steps, we can also show that B_{n+1}^{−1} based on Eq. (2.96) is positive definite.

As examples, Figs. 2.9, 2.10 and 2.11 show respectively the search path of the SR1, DFP, and BFGS methods when applied to find the minimum of the Rosenbrock function.

2.6 Conjugate gradient method

The gradient descent method can be used to solve the minimization problem when the Hessian matrix of the objective function is not available. However, this method may be inefficient if it gets into a zigzag search pattern and repeat the same search directions many times. This problem can be avoided by the conjugate gradient (CG) method. If the objective function is quadratic, the CG method converges to the solution in N iterations without repeating any of the directions previously traversed. If the objective function is not quadratic, the CG method can still significantly improve the performance in comparison to the gradient descent method.

Again consider the approximation of the function f(x) to be minimized by the

Figure 2.10 Search Path of DFP based on B^{−1}

Figure 2.11 Search Path of BFGS based on B^{−1}

first three terms of its Taylor series:

f(x) = f(x_{0} + δx) ≈ f(x_{0}) + g_{0}^{T} δx + \frac{1}{2}δx^{T} H_{0}δx  (2.107)

If the Hessian matrix is positive definite, then f(x) has a minimum. If function f(x) is quadratic, the approximation above becomes exact and the function can be written as

f(x) = \frac{1}{2}x^{T} Ax − b^{T} x + c  (2.108)

where A = H is the symmetric Hessian matrix, and its gradient and Hessian can be written as the following respectively:

g(x) = \frac{d}{dx}f(x) = \frac{d}{dx} (\frac{1}{2}x^{T} Ax − b^{T} x + c) = Ax − b  (2.109)

and

H(x) = \frac{d^{2}}{dx^{2}} f(x) = \frac{d}{dx}g = \frac{d}{dx}(Ax − b) = A  (2.110)

Solving g(x) = Ax − b = 0, we get the solution x^{∗} = A^{−1}b, at which the function is minimized to

f(x^{∗}) = \frac{1}{2}(A^{−1}b)^{T} A(A^{−1}b) − b^{T} (A^{−1}b) + c = −\frac{1}{2}b^{T} A^{−1}b + c (2.111)

At the solution x^{∗} that minimizes f(x), its gradient is

g(x^{∗}) = Ax^{∗} − b = 0  (2.112)

We also see that the minimization of the quadratic function f(x) is equivalent to solving a linear equation Ax = b with a symmetric positive definite coefficient matrix A. The CG method considered here can therefore be used for solving both problems.

Conjugate basis vectors

We first review the concept of conjugate vectors (Section A.1.2) which is of essential importance in the CG method. Two vectors u and v are mutually conjugate or A-conjugate to each other with respect to a symmetric matrix A = A^{T}, if they satisfy:

u^{T} Av = (Av)^{T} u = v^{T} Au = 0  (2.113)

Specially, if A = I, the two conjugate vectors become orthogonal to each other, i.e., v^{T} u = 0.

Similar to a set of N orthogonal vectors that can be used as the basis spanning an N-D space, a set of N mutually conjugate vectors {d_{0}, ⋯, d_{N−1}} satisfying d_{i}^{T} Ad_{j} = 0 (i ≠ j) can also be used as a basis to span the N-D space. Any vector in the space can be expressed as a linear combination of these basis vectors.

Also we note that any set of N independent vectors can be converted by the Gram-Schmidt process (Section A.1.2) to a set of N basis vectors that are either orthogonal or A-conjugate to each other.

Example 2.6 Given two independent basis vectors of the 2-D space, and a positive-definite matrix:

v_{1} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , v_{2} = [ \begin{matrix}0 \\ 1 \end{matrix} ] , A = [ \begin{matrix}3 & 1 \\ 1 & 2 \end{matrix} ]

Figure 2.12 A-Orthogonality based on Matrix A

we can construct two A-orthogonal basis vectors by the Gram-Schmidt method, as shown in Fig. 2.12.

u_{1} = v_{1} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , u_{2} = v_{2} − \frac{u_{1}^{T} Av_{2}}{u_{1}^{T} Au_{1}} u_{1} = [ \begin{matrix}−1/3 \\ 1 \end{matrix} ]

The projections of a vector x = [2, 3]^{T} onto v_{1} and v_{2} are:

p_{v_{1}}(x) = \frac{v_{1}^{T} x}{v_{1}^{T} v_{1}} v_{1} = 2 [ \begin{matrix}1 \\ 0 \end{matrix} ] = [ \begin{matrix}2 \\ 0 \end{matrix} ] , p_{v_{2}}(x) = \frac{v_{2}^{T} x}{v_{2}^{T} v_{2}} v_{2} = 3 [ \begin{matrix}0 \\ 1 \end{matrix} ] = [ \begin{matrix}0 \\ 3 \end{matrix} ]

The A-projections of the same vector x onto u_{1} and u_{2} are:

p_{u_{1}}(x) = \frac{u_{1}^{T} Ax}{u_{1}^{T} Au_{1}} u_{1} = 3 [ \begin{matrix}1 \\ 0 \end{matrix} ] = [ \begin{matrix}3 \\ 0 \end{matrix} ] , p_{u_{2}}(x) = \frac{u_{2}^{T} Ax}{u_{2}^{T} Au_{2}} u_{2} = 3 [ \begin{matrix}−1/3 \\ 1 \end{matrix} ] = [ \begin{matrix}−1 \\ 3 \end{matrix} ]

The original vector x can be represented in either of the two bases:

x = [ \begin{matrix}2 \\ 3 \end{matrix} ] = 2 [ \begin{matrix}1 \\ 0 \end{matrix} ] + 3 [ \begin{matrix}0 \\ 1 \end{matrix} ] = 2v_{1} + 3v_{2} = p_{v_{1}}(x) + p_{v_{2}}(x)
= 3 [ \begin{matrix}1 \\ 0 \end{matrix} ] + 3 [ \begin{matrix}−1/3 \\ 1 \end{matrix} ] = 3u_{1} + 3u_{2} = p_{u_{1}}(x) + p_{u_{2}}(x)

Search along a conjugate basis

Similar to the gradient descent method, which iteratively improves the estimated solution by following a sequence of orthogonal search directions {d_{0}, ⋯, d_{n}, ⋯ } with d_{n} = −g_{n} satisfying g_{n}^{T} g_{n+1} = 0, the CG method also follows a sequence of search directions {d_{0}, ⋯, d_{N−1}} A-orthogonal to each other, i.e., d_{i}^{T} Ad_{j} = 0 (i ≠ j):

x_{n+1} = x_{n} + δ_{n}d_{n} = ⋯ = x_{0} + ∑_{i=0}^{n} δ_{i}d_{i}  (2.114)

We define the error at the nth step as e_{n} = x_{n} − x^{∗}. Then subtracting x^{∗} from both sides, we get the iteration in terms of the errors:

e_{n+1} = e_{n} + δ_{n}d_{n} = ⋯ = e_{0} + ∑_{i=0}^{n} δ_{i}d_{i}  (2.115)

Due to g = Ax − b in Eq. (2.112), we can find the gradient at the nth step x_{n} as

g_{n} = Ax_{n} − b = A(x^{∗} + e_{n}) − b = Ax^{∗} − b + Ae_{n} = Ae_{n}  (2.116)

As the gradient at the solution is zero g(x^{∗}) = 0, we can consider the gradient g_{n} at x_{n} as the residual of the nth iteration, and ε = ||g_{n}||^{2} an error measurement representing how close x_{n} is to the true solution x^{∗}.

The optimal step size given in Eq. (2.39) can now be written as

δ_{i} = −\frac{d_{i}^{T} g_{i}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} Ae_{i}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} A (e_{0} + ∑_{j=0}^{i−1} δ_{j}d_{n})}{d_{i}^{T} Ad_{i}}
= −\frac{d_{i}^{T} Ae_{0} + ∑_{j=0}^{n−1} δ_{j}d_{i}^{T} Ad_{j}}{d_{i}^{T} Ad_{i}} = −\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}}  (2.117)

The last equality is due to the fact that d_{i} and d_{j} are A-orthogonal, d_{i}^{T} Ad_{j} = 0. Substituting this into Eq. (2.115) we get

e_{n+1} = e_{n} + δ_{n}d_{n} = e_{n} − (\frac{d_{n}^{T} Ae_{0}}{d_{n}^{T} Ad_{n}} ) d_{n}  (2.118)

On the other hand, we can represent the error e_{0} = x_{0} − x^{∗} associated with the initial guess x_{0} as a linear combination of the A-orthogonal search vectors {d_{0}, ⋯, d_{N−1}} as N basis vectors that span the N-D vector space:

e_{0} = ∑_{i=0}^{N−1} c_{i}d_{i} = ∑_{i=0}^{N−1} p_{d_{i}}(e_{0}) = ∑_{i=0}^{N−1} (\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.119)

where p_{d_{i}}(e_{0}) is the A-projection of e_{0} onto the ith basis vector d_{i}:

p_{d_{i}}(e_{0}) = c_{i}d_{i} = (\frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.120)

Note that the coefficient c_{i} happens to be the negative optimal step size in Eq. (2.117):

c_{i} = \frac{d_{i}^{T} Ae_{0}}{d_{i}^{T} Ad_{i}} = −δ_{i}  (2.121)

Now the expression of e_{n+1} in Eq. (2.115) can be written as

e_{n+1} = e_{0}+∑_{i=0}^{n} δ_{i}d_{i} = ∑_{i=0}^{N−1} c_{i}d_{i}−∑_{i=0}^{n} c_{i}d_{i} = ∑_{i=n+1}^{N−1} c_{i}d_{i} = ∑_{i=n+1}^{N−1} p_{d_{i}}(e_{0}) (2.122)

We see that in each iteration, the number of terms in the summation for e_{0} is

Figure 2.13 Conjugate Gradient Descent

reduced by one, i.e., the nth component p_{d_{n}}(e_{0}) of e_{0} along the direction of d_{n} is completely eliminated. After N such iterations, the error is reduced from e_{0} to e_{N} = 0, and the true solution is obtained x_{N} = x^{∗} + e_{N} = x^{∗}.

Pre-multiplying d_{k}^{T} A (k ≤ n) on both sides of the equation above, we get

d_{k}^{T} Ae_{n+1} = ∑_{i=n+1}^{N−1} c_{i}d_{k}^{T} Ad_{j} = 0  (2.123)

We see that after n + 1 iterations the remaining error e_{n+1} is A-orthogonal to all previous directions d_{0}, ⋯, d_{n}.

Due to Eq. (2.116), the equation above can also be written as

d_{k}^{T} Ae_{n+1} = d_{k}^{T} g_{n+1} = 0  (2.124)

i.e., the gradient g_{n+1} is orthogonal to all previous search directions.

The conjugate gradient method is compared with the gradient descent method for the case of N = 2 in Fig. 2.13. We see that the first search direction is the same −g_{0} for both methods, but the next search direction d_{1} is A-orthogonal to d_{0}, same as the next error e_{1}, different from the search direction −g_{1} in gradient descent method. The conjugate gradient method finds the solution x in N = 2 steps, while the gradient descent method has to go through many more steps all orthogonal to each other before it finds the solution.

Find the A-orthogonal basis

The N A-orthogonal search directions {d_{0}, ⋯, d_{N−1}} can be constructed based on any set of N independent vectors {v_{0}, ⋯, v_{N−1}} by the Gram-Schmidt process:

d_{n} = v_{n} − ∑_{j=0}^{n−1} p_{d_{j}}(v_{n}) = v_{n} − ∑_{m=0}^{n−1} (\frac{d_{m}^{T}Av_{n}}{d_{m}^{T}Ad_{m}} ) d_{m} = v_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m} (2.125)

where β_{nm} = d_{m}^{T}Av_{n}/(d_{m}^{T}Ad_{m}) and β_{nm}d_{m} is the A-projection of v_{n} onto each of the previous direction d_{m}.

We will gain some significant computational advantage if we choose to use v_{n} = −g_{n}. Now the Gram-Schmidt process above becomes:

d_{n} = −g_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m},  (2.126)

where

β_{nm} = \frac{d_{m}^{T}Av_{n}}{d_{m}^{T}Ad_{m}} = −\frac{d_{m}^{T}Ag_{n}}{d_{m}^{T}Ad_{m}} (m < n)  (2.127)

The equation above indicates that g_{n} can be written as a linear combination of all previous search directions d_{0}, ⋯, d_{n}:

g_{n} = ∑_{i=0}^{n} α_{i}d_{i}  (2.128)

Pre-multiplying g_{n+1}^{T} on both sides, we get

g_{n+1}^{T}g_{k} = g_{n+1}^{T} (∑_{i=0}^{k} α_{i}d_{i}) = ∑_{i=0}^{k} α_{i} g_{n+1}^{T}d_{i} = 0  (2.129)

The last equality is due to g_{n+1}^{T}d_{k} = 0 in Eq. (2.124). We see that g_{n+1} is also orthogonal to all previous gradients g_{0}, ⋯, g_{n}.

Pre-multiplying g_{k}^{T} (k ≥ n) on both sides of Eq. (2.126), we get

g_{k}^{T} d_{n} = −g_{k}^{T} g_{n} − ∑_{m=0}^{n−1} β_{mn} g_{k}^{T} d_{m} = −g_{k}^{T} g_{n} = \{ \begin{matrix}−||g_{n}||^{2} & n = k \\ 0 & n < k \end{matrix} (2.130)

Note that all terms in the summation are zero as g_{k}^{T} d_{m} = 0 for all m < n ≤ k (Eq. (2.124)). Substituting g_{n}^{T} d_{n} = −||g_{n}||^{2} into Eq. (2.117), we get

δ_{n} = −\frac{g_{n}^{T} d_{n}}{d_{n}^{T} Ad_{n}} = \frac{||g_{n}||^{2}}{d_{n}^{T} Ad_{n}}  (2.131)

Next we consider

g_{m+1} = Ax_{m+1}−b = A(x_{m}+δ_{m}d_{m})−b = (Ax_{m}−b)+δ_{m}Ad_{m} = g_{m}+δ_{m}Ad_{m}

(2.132)

Pre-multiplying g_{n}^{T} with n > m on both sides we get

g_{n}^{T} g_{m+1} = g_{n}^{T} g_{m} + δ_{m}g_{n}^{T} Ad_{m} = δ_{m}g_{n}^{T} Ad_{m}  (2.133)

where g_{n}^{T} g_{m} = 0 (m ≠ n). Solving for g_{n}^{T} Ad_{m} we get

g_{n}^{T} Ad_{m} = \frac{1}{δ_{m}} g_{n}^{T} g_{m+1} = \{ \begin{matrix}||g_{n}||^{2}/δ_{n−1} & m = n − 1 \\ 0 & m < n − 1 \end{matrix}  (2.134)

Substituting this into Eq. (2.127) we get

β_{nm} = −\frac{d_{m}^{T}Ag_{n}}{d_{m}^{T}Ad_{m}} = \{ \begin{matrix}−||g_{n}||^{2}/δ_{n−1}d_{n−1}^{T}Ad_{n−1} & m = n − 1 \\ 0 & m < n − 1 \end{matrix}  (2.135)

which is non-zero only when m = n − 1, i.e., there is only one non-zero term in the summation of the Gram-Schmidt formula for d_{n}. This is the reason why we choose v_{n} = −g_{n}. We can now drop the second subscript m in β_{nm}, and Eq. (2.125) becomes:

d_{n} = v_{n} − ∑_{m=0}^{n−1} β_{nm}d_{m} = −g_{n} − β_{n}d_{n−1}  (2.136)

Substituting the step size δ_{n−1} = ||g_{n−1}||^{2}/d_{n−1}^{T}Ad_{n−1} (Eq. (2.131)) into the above expression for β_{nm} = β_{m}, we get

β_{n} = −\frac{||g_{n}||^{2}}{||g_{n−1}||^{2}}  (2.137)

We note that matrix A no longer appears in the expression.

The CG algorithm

Summarizing the above, we finally get the conjugate gradient algorithm in the following steps:

1. Set n = 0 and initialize the search direction (same as gradient descent):

d_{0} = −g_{0}  (2.138)

2. Terminate if the error ε = ||g_{n}||^{2} is smaller than a preset threshold. Otherwise, continue with the following:

3. Find optimal step size (Eq. (2.131)) and step forward

δ_{n} = \frac{||g_{n}||^{2}}{d_{n}^{T} Ad_{n}} ,  x_{n+1} = x_{n} + δ_{n}d_{n}  (2.139)

4. Update gradient:

g_{n+1} = \frac{d}{dx}f(x_{n+1})  (2.140)

5. Find coefficient for the Gram-Schmidt process (Eq. (2.137):

β_{n+1} = −\frac{||g_{n+1}||^{2}}{||g_{n}||^{2}}  (2.141)

6. Update search direction (Eq. (2.136)):

d_{n+1} = −g_{n+1} − β_{n+1}d_{n}  (2.142)

Set n = n + 1 and go back to step 2.

The algorithm above assumes the objective function f(x) to be quadratic with known A. But when f(x) is not quadratic, A is no longer available, we can still approximate f(x) as a quadratic function by the first three terms in its Taylor series in the neighborhood of its minimum. Also, the we need to modify the algorithm so that it does not depend on A. Specifically, the optimal step size δ_{n} calculated in step 3 above based on A can also be alternatively found by line minimization based on any suitable algorithms for 1-D optimization.

The Matlab code for the conjugate gradient algorithm is listed below:

function xn=myCG(j,tol)       % j is the objective function to minimize
syms d;                   % variable for 1-D symbolic function f(d)
x=symvar(j).';            % symbolic variables in objective function
J=matlabFunction(j);      % the objective function
G=jacobian(j).';          % symbolic gradient of objective function
G=matlabFunction(G);      % the gradient function
xn=zeros(length(x),1);    % initial guess of x
xc=num2cell(xn);
e=J(xc{:});               % initial error
gn=G(xc{:});              % gradient at xn
dn=-gn;                   % use negative gradient as search direction
n=0;
while e>tol
    n=n+1;
    f=subs(j,x,xn+d*dn);  % convert n-D f(x) to 1-D f(d)
    delta=Opt1d(f);       % find delta that minimizes 1D function f(d)
    xn=xn+delta*dn;       % updata variable x
    xc=num2cell(xn);
    e=J(xc{:});           % new error
    gn1=G(xc{:});         % new gradient
    bt=-(gn1.'*gn1)/(gn.'*gn);    % find beta
    dn=-gn1-bt*dn;        % new search direction
    gn=gn1;               % update gradient
    fprintf('%d\t(%.4f, %.4f, %.4f)\t%e\n',n,xn(1),xn(2),xn(3),e)
end
end

Here is the function that uses Newton’s method to find the optimal step size δ that minimizes the objective function as a 1-D function of δ:

function x=Opt1d(f)         % f is 1-D symbolic function to minimize
syms x;

tol=10^(-3);
d1=diff(f);             % 1st order derivative
d2=diff(d1);            % 2nd order derivative
f=matlabFunction(f);
d1=matlabFunction(d1);
d2=matlabFunction(d2);
x=0.0;                  % initial guess of delta
if d2(x)<=0             % 2nd order derivative needs to be positive
    x=rand-0.5;e
end
y=x+1;
while abs(x-y) > tol    % minimization
    y=x;
    x=y-d1(y)/d2(y);    % Newton iteration
end
end

Example 2.7 To compare the conjugate method and the gradient descent method, consider a very simple 2-D quadratic function

f(x, y) = x^{T} Ax = [x_{1}, x_{2}] [ \begin{matrix}3 & 1 \\ 1 & 2 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]

The performance of the gradient descent method depends significantly on the initial guess. For the specific initial guess of x_{0} = [1.5, −0.75]^{T}, the iteration gets into a zigzag pattern and the convergence is very slow, as shown below:

\begin{matrix}n & x = [x_{1}, x_{2}] & f(x) \\ 0 & 1.500000, −0.750000 & 2.812500 \\ 1 & 0.250000, −0.750000 & 0.468750e − 01 \\ 2 & 0.250000, −0.125000 & 7.812500e − 02 \\ 3 & 0.041667, −0.125000 & 1.302083e − 02 \\ 4 & 0.041667, −0.020833 & 2.170139e − 03 \\ 5 & 0.006944, −0.020833 & 3.616898e − 04 \\ 6 & 0.006944, −0.003472 & 6.028164e − 05 \\ 7 & 0.001157, −0.003472 & 1.004694e − 05 \\ 8 & 0.001157, −0.000579 & 1.674490e − 06 \\ 9 & 0.000193, −0.000579 & 2.790816e − 07 \\ 10 & 0.000193, −0.000096 & 4.651361e − 08 \\ 11 & 0.000032, −0.000096 & 7.752268e − 09 \\ 12 & 0.000032, −0.000016 & 1.292045e − 09 \\ 13 & 0.000005, −0.000016 & 2.153408e − 10 \end{matrix}

However, as expected, the conjugate gradient method takes exactly N = 2 steps from any initial guess to reach at the solution:

Figure 2.14 Gradient (left) vs. Conjugate Gradient (right)

\begin{matrix}n & x = [x_{1}, x_{2}] & f(x) \\ 0 & 1.500000, −0.750000 & 2.812500e + 00 \\ 1 & 0.250000, −0.750000 & 4.687500e − 01 \\ 2 & 0.000000, −0.000000 & 1.155558e − 33 \end{matrix}

In Fig. 2.14, the search paths of these two methods are compared.

For an N = 3 example of f(x) = x^{T} Ax with

A = [ \begin{matrix}5 & 3 & 1 \\ 3 & 4 & 2 \\ 1 & 2 & 3 \end{matrix} ]

from an initial guess x_{0} = [1, 2, 3]^{T}, it takes the gradient descent method 41 iterations to reach x_{41} = [3.5486e − 06, −7.4471e − 06, 4.6180e − 06]^{T} corresponding to f(x) = 8.5429e − 11. From the same initial guess, it takes the conjugate gradient method only N = 3 iterations to converge to the solution:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & f(x) \\ 0 & 1.000000, 2.000000, 3.000000 & 4.500000e + 01 \\ 1 & −0.734716, −0.106441, 1.265284 & 2.809225e + 00 \\ 2 & 0.123437, −0.209498, 0.136074 & 3.584736e − 02 \\ 3 & −0.000000, 0.000000, 0.000000 & 3.949119e − 31 \end{matrix}

For an N = 9 example, it takes over 4000 iterations for the gradient descent method to converge with ||e|| ≈ 10^{−10}, but exactly 9 iterations for the CG method to converge with ||e|| ≈ 10^{−16}.

Example 2.8 The search path of the conjugate gradient method applied to the minimization of the Rosenbrock function is shown in Fig. 2.15.

Figure 2.15 Conjugate Gradient Applied to Rosenbrock Surface

Example 2.9 Solve the following N = 3 non-linear equation system by the CG method:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

The solution is known to be x_{1} = 0.5, x_{2} = 0, x_{3} = −0.5.

This equation system can be represented in vector form as f(x) = 0 and the objective function is J(x) = f^{T} (x)f(x). The iteration of the CG method with an initial guess x_{0} = 0 is shown below:

\begin{matrix}n & x = [x_{1}, x_{2}, x_{3}] & J(x) \\ 0 & 0.0000, 0.0000, 0.0000 & 1.032500e + 02 \\ 1 & 0.0113, 0.0050, −0.5001 & 3.160163e + 00 \\ 2 & 0.0188, −0.0021, −0.5004 & 3.095894e + 00 \\ 3 & 0.5009, −0.0018, −0.5004 & 7.268252e − 05 \\ 4 & 0.5009, −0.0017, −0.5000 & 1.051537e − 05 \\ 5 & 0.5008, −0.0012, −0.5000 & 6.511151e − 06 \\ 6 & 0.5001, −0.0005, −0.5000 & 6.365321e − 07 \\ 7 & 0.5001, −0.0005, −0.5000 & 5.667357e − 07 \\ 8 & 0.5002, −0.0004, −0.5000 & 2.675128e − 07 \\ 9 & 0.5001, −0.0003, −0.5000 & 1.344218e − 07 \\ 10 & 0.5001, −0.0002, −0.5000 & 1.241196e − 07 \\ 11 & 0.5000, −0.0001, −0.5000 & 2.120969e − 08 \\ 12 & 0.5000, −0.0001, −0.5000 & 1.541814e − 08 \\ 13 & 0.5000, −0.0001, −0.5000 & 7.282025e − 09 \\ 14 & 0.5000, −0.0001, −0.5000 & 4.801781e − 09 \\ 15 & 0.5000, −0.0000, −0.5000 & 4.463926e − 09 \end{matrix}

In comparison, the gradient descent method would need to take over 200 iterations (with much reduced complexity though) to reach this level of error.

Conjugate gradient method used for solving linear equation systems

As discussed before, if x is the solution that minimizes the quadratic function f(x) = x^{T} Ax/2 − b^{T} x + c, with A = A^{T} being symmetric and positive definite, it also satisfies d f(x)/dx = Ax − b = 0. In other words, the optimization problem is equivalent to the problem of solving the linear system Ax − b = 0, both can be solved by the conjugate gradient method.

Now consider solving the linear system Ax = b with A = A^{T}. Let d_{i}, (i = 1, ⋯, N) be a set of N A-orthogonal vectors satisfying d_{i}^{T} Ad_{j} = 0 (i ≠ j), which can be generated based on any N independent vectors, such as the standard basis vectors, by the Gram-Schmidt method. The solution x of the equation Ax = b can be represented by these N vectors as

x = ∑_{i=1}^{N} c_{i}d_{i}  (2.143)

Now we have

b = Ax = A [∑_{i=1}^{N} c_{i}d_{i}] = ∑_{i=1}^{N} c_{i}Ad_{i}  (2.144)

Pre-multiplying d_{j}^{T} on both sides we get

d_{j}^{T} b = ∑_{i=1}^{N} c_{i}d_{j}^{T} Ad_{i} = c_{j}d_{j}^{T} Ad_{j}  (2.145)

Solving for c_{j} we get

c_{j} = \frac{d_{j}^{T} b}{d_{j}^{T} Ad_{j}}  (2.146)

Substituting this back into the expression for x we get the solution of the equation:

x = ∑_{i=1}^{N} c_{i}d_{i} = ∑_{i=1}^{N} (\frac{d_{i}^{T} b}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.147)

Also note that as b = Ax, the ith term of the summation above is simply the A-projection of x onto the ith direction d_{i}:

p_{d_{i}}(x) = (\frac{d_{i}^{T} Ax}{d_{i}^{T} Ad_{i}} ) d_{i}  (2.148)

One application of the conjugate gradient method is to solve the normal equation to find the least-square solution of an over-constrained equation system Ax = b, where the coefficient matrix A is M by N with rank R = N < M. As discussed previously, the normal equation of this system is

A^{T} Ax = A^{T} b  (2.149)

Here A^{T} A is an N by N symmetric, positive definite matrix. This normal equation can be solved by the conjugate gradient method.

2.7 Issues of Local/Global Minimum

A main concern in optimization problems is whether the solution obtained by a certain algorithm is a local minimum or a global one. If function f(x) to be optimized is quadratic and definite (either positive or negative definite), then any local extremum (either minimum or maximum) is the unique and global extremum. However, if f(x) is not quadratic, then a local minimum found by the algorithm may not be global. In such a case, the simulated annealing (SA) algorithm can be used to search for the global minimum, by allowing up-hill moves in the iteration to avoid getting stuck at a local minimum. SA mimics the annealing process in metallurgy, in which the temperature is carefully controlled to decrease slowly so that the material reaches a state with minimum free energy.

The way the SA method avoids getting stuck at a local minimum is to allow the iteration steps to go up-hill. Specifically, the objective function f(x) evaluated at x_{n} is treated as the energy at x_{n}, and an up-hill move corresponding to an energy increase ΔE = f(x_{n+1}) − f(x_{n}) can be accepted if some probability function P (ΔE, T) defined below is greater than a threshold value Th within a certain range comparable with P:

P (ΔE, T) = e^{−ΔE/T} > Th  (2.150)

where T represents the temperature. Obviously smaller ΔE and higher T will cause higher P, it is more probable for a move to be accepted.

During the entire search process, the temperature T gradually reduces so that the probability of accepting an up-hill move also gradually reduces. In the early stage of the search, it is more probable to go up-hill, thereby getting out of a local minimum; but toward the later part of the search process, such a move is less likely to happen, as presumably in this late stage of the search, the energy level is already low, not likely to be at a local minimum with high energy level. Through out the search, the best result corresponding to he lowest energy level is always updated, to be returned to as the final solution at the end of the process.

Problems

Develop Matlab code (or any other language) to implement the algorithms covered in this chapter to solve the equations in the examples in the text. Try at least three different initial guesses to see how they may affect how quickly the iteration converges.

1. Consider the linear over-constrained system in the form Ax = b, where A is a M by N matrix, x and b are respectively N-D and M-D vectors. Specifically, let M = 9 and N = 5, and generate in Matlab by rand(′seed′, 1), A = rand(M, N) and b = rand(M, 1), and then find the optimal solution of equation system so that the LS error ε = ||Ax − b||^{2} is minimized.

2. Plot the following single-variable non-linear functions. Make sure the domain of the function in the plot is wide enough to cover at least one root (zerocrossing):

f_{1}(x) = e^{x} − 1/x, f_{2}(x) = sin(x) − x^{3}, f_{3}(x) = x^{3} + x^{2} − 26 cos(x)

Then find their roots by solving the corresponding equations by the Newton-Raphson method.

3. Solve the same equations in the previous problem by minimizing the objective function defined as o(x) = f^{2}(x) by Newton’s method. Try at least three different initial guesses and compare results.

4. Solve the following equation systems by the multivariable Newton-Raphson method. Try at least three different initial guesses and compare results in terms of convergence of the iteration.

\{ \begin{matrix}f_{1}(x) = x_{1}^{2} − 2x_{1} + x_{2}^{2} − x_{3} + 1 = 0 \\ f_{2}(x) = x_{1}x_{2}^{2} − x_{1} − 3x_{2} + x_{2}x_{3} + 2 = 0 \\ f_{3}(x) = x_{1}x_{3}^{3} − 3x_{3} + x_{2}x_{3}^{2} + x_{1}x_{2} = 0 \end{matrix}  \{ \begin{matrix}f_{1}(x) = 3x − 1 − (x_{2}x_{3})^{2} − 3/2 = 0 \\ f_{2}(x) = 4x_{1}^{2} − 500x_{2}^{2} + 2x_{2} − 1 = 0 \\ f_{3}(x) = exp(−x_{1}x_{2}) + 20x_{3} + 9 = 0 \end{matrix}

5. Resolve the equation systems in the previous problem in vector form f(x) = 0, by minimizing the objective function defined as o(x) = ||f(x)||^{2}) = f^{T} f, using Newton’s method for minimizing multivariable functions. Try at least three different initial guesses and compare results.

6. Make a 3-D surface plot and a 2-D contour plot of the Rosenbrock function

f(x, y) = (1 − x)^{2} + 100(y − x^{2})^{2}

Use both the gradient descent method (without second order derivatives) and Newton’s method to find the minimum of the function from different initial guesses such as: [−1, 2], [−1, 0], [0, 1], [2, 2]. Plot the search paths, same as in Fig. 2.7

7. Resolve the same problem above but using various versions of quasi-Newton’s method (SR1, BFGS, DFP). Try the same initial guesses and plot the search paths.

8. Resolve the same problem above but using the conjugate gradient method. Try the same initial guesses and plot the search paths.

9. Solve the following equation system by the conjugate gradient method:

\{ \begin{matrix}f_{1}(x_{1}, x_{2}, x_{3}) = 3x_{1} − (x_{2}x_{3})^{2} − 3/2 \\ f_{2}(x_{1}, x_{2}, x_{3}) = 4x_{1}^{2} − 625 x_{2}^{2} + 2x_{2} − 1 \\ f_{3}(x_{1}, x_{2}, x_{3}) = exp(−x_{1}x_{2}) + 20x_{3} + 9 \end{matrix}

3 Constrained Optimization

This chapter for constrained optimization contains some theoretically intensive discussions about constrained optimization, including theories and methods for optimization with both equality and inequality constraints, and the KKT conditions for both primal and dual problems of constrained optimization. Such discussions are necessary for the study of certain topics in future chapters, such as the method of support vector machines, one of the most important classification method in machine learning, to be considered in Chapter 14.

An optimization problem becomes more complicated if it is constrained, i.e., the arguments x of the objective function f(x) are subject to (abbreviated as s.t. in the following) certain equality and/or inequality constraints in terms of what values they can take. Such a generic constrained optimization problem can be formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & \{ \begin{matrix}h_{i}(x) = 0, (i = 1, ⋯ m) \\ g_{j}(x) ≤ 0, (j = 1, ⋯ n) \end{matrix} \end{matrix}  (3.1)

The set composed of all values of x that satisfy the m + n constraints is called the feasible region of the problem. The goal is to find a point x^{∗} in the feasible region at which f(x^{∗}) is an extremum.

In the following we will first consider the most general case where the objective function f(x) is nonlinear and so are the constraint functions g_{i}(x) and h_{j}(x). The processing of solving such a problem is called the nonlinear programming (NLP). Here programming means the mathematical procedure for solving the optimization problems. We will consider the general method of Lagrange multiplier and a set of conditions all solutions of such problems must satisfy.

We will then consider some specific algorithms for two special cases: linear programming (LP) in Section 3.4 for problems where the objective function and all constraint functions are linear (the feasible region is a polytope), and then quadratic programming (QP) in Section 3.6 for problems where the objective function is quadratic while the constraints are linear.

Figure 3.1 Equality Constrained Optimization of Function f(x, y)

3.1 Optimization with Equality Constraints

The optimization problems subject to equality constraints can be generally formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & h_{i}(x) = h_{i}(x_{1}, ⋯, x_{N}) = 0, (i = 1, ⋯ m) \end{matrix}  (3.2)

Here we assume m is no bigger than N, as in general there does not exist a solution that satisfies more than N equations in the N-D space.

This problem can be visualized in the special case with N = 2 and m = 1, where both f(x) and h(x) are surfaces defined over the 2-D space spanned by x_{1} and x_{2}, and h(x) = 0 is the intersection line of h(x) and the 2-D plane. The solution x^{∗} must satisfy the following two conditions:

• x^{∗} is on the intersection line satisfying h(x^{∗}) = 0;

• x^{∗} is on the contour line f(x) = d (for some d as shown in Fig. 3.1, so that f(x^{∗}) will neither increase nor decrease while moving along the contour line in the neighborhood of x^{∗}.

For both conditions to be satisfied, the two contours h(x) = 0 and f(x) = d must coincide at x^{∗}, i.e., they must have the same tangent line and therefore parallel gradients (perpendicular to the tangents):

▽_{x}f(x^{∗}) = λ ▽_{x} h(x^{∗})  (3.3)

where ▽_{x}f(x) = [∂f/∂x_{1}, ⋯, ∂f/∂x_{N}]^{T} is the gradient of f(x), and λ is a constant scaling factor which can be either positive if the two gradients are in the same direction or negative if they are in opposite directions. As here we are only interested in whether the two curves coincide or not, the sign of λ can be either positive or negative.

To find such a point x^{∗}, we first construct a new objective function, called the Lagrange function (or simply Lagrangian):

L(x, λ) = f(x) − λ h(x)  (3.4)

where λ is the Lagrange multiplier, which can be either positive or negative (i.e., the sign in front of λ can be either plus or minus), and then set the gradient of this objective function with respect to λ as well as x to zero:

▽_{x,λ}L(x, λ) = ▽_{x,λ}[f(x) − λ h(x)] = 0  (3.5)

This equation contains two parts:

\{ \begin{matrix}▽_{x}f(x) = λ ▽_{x} h(x) \\ ▽_{λ}L(x, λ) = ∂L(x, λ)/∂λ = −h(x) = 0 \end{matrix}  (3.6)

by which both Eq. (3.3) and the equality constraint are satisfied as desired.

This is an equation system containing N + m = 2 + 1 = 3 equations for the same number of unknowns x_{1}, x_{2}, λ:

\frac{∂f(x)}{∂x_{i}} = λ\frac{∂h(x)}{∂x_{i}} (i = 1, 2), h(x) = 0  (3.7)

The solution x^{∗} = [x_{1}^{∗}, x_{2}^{∗}]^{T} and λ^{∗} of this equation system happens to satisfy the two desired relationships: (a) ▽_{x}f(x) = λ ▽_{x} h(x), and (b) h(x^{∗}) = 0.

Specially, consider the following two possible cases:

• If λ^{∗} = 0, then

▽_{x}f(x^{∗}) = λ^{∗} ▽_{x} h(x^{∗}) = 0  (3.8)

indicating f(x^{∗}) is an extremum independent of the constraint, i.e., the constraint h(x) = 0 is inactive, and the optimization is unconstrained.

• If λ^{∗} ≠ 0, then ▽_{x}f(x^{∗}) ≠ 0 (as in general ▽_{x}h(x^{∗}) ≠ 0), indicating f(x^{∗}) is not an extremum without the constraint, i.e., the constraint is active, and the optimization is indeed constrained.

The discussion above can be generalized from 2-D to any N > 2 dimensional space, in which the optimal solution x^{∗} is to be found to extremize the objective f(x) subject to m equality constraints h_{i}(x) = 0 (i = 1, ⋯, m), each representing a contour surface in the N-D space. The solution x^{∗} must satisfy the following two conditions:

• x^{∗} is on all m contour surfaces to satisfy h_{1}(x^{∗}) = ⋯ = h_{m}(x^{∗}) = 0, i.e., it is on the intersection of these surfaces, a curve in the space (e.g., the intersection of two surfaces in 3-D space is a curve);

• x^{∗} is on a contour surface f(x^{∗}) = d (for some value d) so that f(x^{∗}) is an extremum, i.e., it does not increase or decrease while moving along the curve in the neighborhood of x^{∗}.

Combining these conditions, we see that the intersection (a curve or a surface) of the m surfaces h_{i}(x) = 0 must coincide with the contour surface f(x) = d

Figure 3.2 Derivation of Lagrange Function

at x^{∗}, i.e., the intersection of the tangent surfaces of h_{i}(x) = 0 through x^{∗} must coincide with the tangent surface of f(x) = d at x^{∗}, therefore the gradient ▽f(x^{∗}) must be a linear combination of the m gradients ▽h_{i}(x^{∗}):

▽_{x}f(x^{∗}) = ∑_{i=1}^{m} λ_{i}^{∗} ▽_{x} h_{i}(x^{∗})  (3.9)

as shown in Fig. 3.2.

To find such a solution x^{∗} of the optimization problem, we first construct the Lagrange function:

L(x, λ) = f(x) − ∑_{i=1}^{m} λ_{i}h_{i}(x)  (3.10)

where λ = [λ_{1}, ⋯, λ_{m}]^{T} is a vector for m Lagrange multipliers. We then set its gradient to zero:

▽_{x,λ}L(x, λ) = ▽_{x,λ} [f(x) − ∑_{i=1}^{m} λ_{i}h_{i}(x)] = 0  (3.11)

to get the following two equation systems of N and m equations respectively:

▽_{x}f(x) = ∑_{i=1}^{m} λ_{i} ▽_{x} h_{i}(x) i.e. \frac{∂f(x)}{∂x_{j}} = ∑_{i=1}^{m} λ_{i} \frac{∂h_{i}(x)}{∂x_{j}} (j = 1, ⋯, N),

(3.12)

and

\frac{∂L(x, λ)}{∂λ_{i}} = h_{i}(x) = 0 (i = 1, ⋯, m)  (3.13)

The first set of N equations indicates that the gradient ▽_{x}f(x^{∗}) at x^{∗} is a linear combination of the m gradients ▽_{x}h_{i}(x^{∗}), while the second set of m equations

Figure 3.3 Inequality Constrained Optimization of f(x, y)

guarantees that x^{∗} also satisfy the m equality constraints, both as desired. Solving these equations we get the desired solution x^{∗} together with the Lagrange multipliers λ_{1}, ⋯, λ_{m}. This is the Lagrange multiplier method, by which the constrained optimization problem is reformulated in such a way that the solution can be found by solving the equation system obtained by setting the gradient of the Lagrange function to zero.

Specially, if λ_{i} = 0, then the ith constraint is not active. More specially if λ_{i} = 0 for all i = 1, ⋯, m, then

▽_{x}f(x^{∗}) = ∑_{i=1}^{m} λ_{i} ▽_{x} h_{i}(x) = 0  (3.14)

indicating f(x^{∗}) is an extremum independent of any of the constraints, i.e., the optimization problem is unconstrained.

3.2 Optimization with Inequality Constraints

The optimization problems subject to inequality constraints can be generally formulated as:

\{ \begin{matrix}max/min: & f(x) = f(x_{1}, ⋯, x_{N}) \\ s. t.: & g_{j}(x) = g_{j}(x_{1}, ⋯, x_{N}) ≤ or ≥ 0,  (j = 1, ⋯ n) \end{matrix} (3.15)

Again, to visualize the problem we first consider an example with N = 2 and n = 1, as shown in Fig. 3.3 for the minimization (left) and maximization (right) of f(x) subject to g(x) > 0. Here the constrained solution x^{∗} = [x_{1}^{∗}, x_{2}^{∗}]^{T} is on the boundary of the feasible region satisfying g(x_{1}, x_{2}) = 0, while the unconstrained extremum is outside the feasible region.

We consider the following two possible cases.

• First, if the unconstrained extremum is inside the feasible region, i.e., the

Figure 3.4 Determination of Signs of Lagrange Multipliers

inequality constraint is inactive, then the solution x^{∗} is the same as that of an unconstrained problem, not on the boundary of the feasible region, i.e.,

▽_{x}f(x^{∗}) = 0, g(x^{∗}) ≠ 0  (3.16)

Similar to the case of an inactive equality constraint, here the multiplier for an inactive inequality constraint is zero, i.e., μ^{∗} = 0, and we get

▽_{x}f(x^{∗}) = μ^{∗} ▽_{x} g(x^{∗}) = 0  (3.17)

the same as in Eq. (3.8).

• Second, if the unconstrained extremum at which ▽_{x}f(x) = 0 is outside the feasible region, i.e., the inequality constraint is active, then the constrained solution x^{∗} must be on the boundary of the feasible region, and it is different from the unconstrained solution:

g(x^{∗}) = 0,  ▽_{x}f(x^{∗}) ≠ 0  (3.18)

In this case, the problem becomes the same as an equality constrained problem considered before, i.e., the objective function f(x) and the constraint function g(x) have the same tangent at x^{∗} with parallel gradients:

▽_{x}f(x^{∗}) = μ^{∗} ▽_{x} g(x^{∗}) or ▽_{x} f(x^{∗}) − μ^{∗} ▽_{x} g(x^{∗}) = 0 (3.19)

However, different from before, now we also need to consider whether the two gradients have the same or opposite directions, corresponding to μ > 0 or μ < 0, respectively. Specifically, there exist four possible cases in terms of the sign of μ, depending on whether f(x) is to be maximized or minimized, and whether the constraint is g(x) ≥ 0 or g(x) ≤ 0, as the four points x_{1} through x_{4} illustrated in the 1-D examples in Fig. 3.4 and also summarized in the tables below:

g(x) ≥ 0  g(x) ≤ 0
max f(x) (1) ▽ f(x_{1}) = μ ▽ g(x_{1}), μ < 0 (3) ▽ f(x_{3}) = μ ▽ g(x_{3}), μ > 0
min f(x) (2) ▽ f(x_{2}) = μ ▽ g(x_{2}), μ > 0 (4) ▽ f(x_{4}) = μ ▽ g(x_{4}), μ < 0

(3.20)

max/min subject to  μ
x_{1} max f(x) g(x) ≥ 0 μ < 0
x_{2} min f(x) g(x) ≥ 0 μ > 0
x_{3} max f(x) g(x) ≤ 0 μ > 0
x_{4} min f(x) g(x) ≤ 0 μ < 0

(3.21)

We can also consider the product of μ and g(x) for either maximization or minimization:

μ g(x) \{ \begin{matrix}≤ 0 & for maximization \\ ≥ 0 & for minimization \end{matrix}  (3.22)

Summarizing the two cases above, we see that g(x^{∗}) ≠ 0 but μ^{∗} = 0 in the first case of an inactive constraint, and g(x^{∗}) = 0 but μ^{∗} ≠ 0 in the second case of an active constraint, i.e., the following holds in both cases:

μ^{∗} g(x^{∗}) = 0  (3.23)

The discussion above can be generalized from 2-D to N > 2 dimensional space, in which the optimal solution x^{∗} is to be found to extremize the objective f(x) subject to n inequality constraints g_{i}(x) = 0 (i = 1, ⋯, n). To solve this inequality constrained optimization problem, we first construct the Lagrangian:

L(x, μ) = f(x) − ∑_{i=1}^{n} μ_{i} g_{i}(x)  (3.24)

where μ = [μ_{1}, ⋯, μ_{n}]^{T} is a vector for n Lagrange multipliers. (We note that in some literature, a plus sign is used in front of the summation of the second term. This is equivalent to our discussion here so long as the sign of μ indicated in Table 3.21 is negated.) Now Eq. (3.23) can be written as

(μ^{∗})^{T} g(x^{∗}) = 0  (3.25)

We then set the gradient of the Lagrangian to zero:

▽_{x,μ}L(x, μ) = ▽_{x,μ} [f(x) − ∑_{i=1}^{n} μ_{i} g_{i}(x)] = 0  (3.26)

and get two equation systems of N and n equations, respectively:

▽_{x}f(x) = ∑_{i=1}^{n} μ_{i} ▽_{x} g_{i}(x)  (3.27)

and

▽_{μ} [∑_{i=1}^{n} μ_{i} g_{i}(x)] = 0 i.e. \frac{∂L(x, μ)}{∂μ_{i}} = g_{i}(x) = 0 (i = 1, ⋯, n) (3.28)

The result above for the inequality constrained problems based on Eq. (3.26) looks the same as that for the equality constrained problems based on Eq. (3.11), but with an additional requirement regarding the sign of the Lagrange multipliers. For an equality constrained problem, the direction of the gradient ▽g(x) is of no concern, i.e., the sign of λ is unrestricted; but here for an inequality constrained problem, the sign of μ needs to be consistent with those shown in Table 3.21, unless when the constraint is inactive and μ = 0.

As a simple example, consider the minimization of a function f(x) by the method of gradient descent. To find the best increment Δx that minimizes f(x+ Δx) ≈ f(x) + g_{f}(x)^{T} Δx subjected to the constraint of a limited length, i.e., ||Δx||^{2}/2 ≤ C, we first construct the Lagrangian:

L(x, λ) = f(x) + g_{f}(x)^{T} Δx − μ(||Δx||^{2}/2 − C)  (3.29)

and set its derivative to zero:

\frac{d}{dΔx}L(x, λ)g_{f}(x) − μΔx = 0  (3.30)

where μ < 0 according to Table 3.21. We then solve the equation to get

Δx = \frac{1}{μ}g_{f}(x)  (3.31)

and the iteration

x_{n+1} = x_{n} − δg(x_{n})  (3.32)

same as the gradient descent method in Eq. (2.25).

We now consider the general optimization of an N-D objective function f(x) subject to multiple constraints of qualities and inequalities:

max/min: f(x) = f(x_{1}, ⋯, x_{N})
s. t.:  \{ \begin{matrix}h_{i}(x) = 0, (i = 1, ⋯ m) \\ g_{j}(x) ≤ 0 or g_{j}(x) ≥ 0,  (j = 1, ⋯ n) \end{matrix} (3.33)

For notational convenience, we denote the m + n equality and inequality constraints in vector form as:

max/min: f(x)
s. t.:  \{ \begin{matrix}h(x) = 0 \\ g(x) ≤ 0 or g(x) ≥ 0 \end{matrix}

(3.34)

where h(x) = [h_{1}(x), ⋯, h_{m}(x)]^{T} and g(x) = [g_{1}(x), ⋯, g_{n}(x)]^{T}.

To solve this optimization problem, we first construct the Lagrangian

L(x, λ, μ) = f(x)−∑_{i=1}^{m} λ_{i}h_{i}(x)−∑_{j=1}^{n} μ_{j}g_{j}(x) = f(x)−λ^{T} h(x)−μ^{T} g(x) (3.35)

Figure 3.5 Inequality Constrained Optimization

where the Lagrange multipliers in λ = [λ_{1}, ⋯, λ_{m}]^{T} and μ = [μ_{1}, ⋯, μ_{n}]^{T} are for the m equality and n non-negative constraints, respectively. We then set to zero the gradient of L(x, λ, μ) with respect to both λ and μ as well as x.

\frac{∂}{∂x}L(x, λ, μ) = 0, \frac{∂}{∂λ}L(x, λ, μ) = 0, \frac{∂}{∂μ}L(x, λ, μ) = 0  (3.36)

Solving these equation systems we get the solution x^{∗}. While λ_{i} can be either positive or negative, the sign of μ_{j} needs to be consistent with those specified in Table 3.21. Otherwise the inequality constraints is inactive.

Example 3.1 Find the extremum of f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} subject to each of the three different constraints: x_{1} + x_{2} = 1, x_{1} + x_{2} ≤ 1, and x_{1} + x_{2} ≥ 1, as illustrated in Fig. 3.5.

•

\{ \begin{matrix}min: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & h(x_{1}, x_{2}) = x_{1} + x_{2} − 1 = 0 \end{matrix}

The Lagrangian is:

L(x_{1}, x_{2}, λ) = f(x_{1}, x_{2}) − λg(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} − λ(x_{1} + x_{2} − 1)

Solving the following equations

\frac{∂L}{∂x_{1}} = 2x_{1} − λ = 0, \frac{∂L}{∂x_{2}} = 2x_{2} − λ = 0, \frac{∂L}{∂λ} = x_{1} + x_{2} − 1 = 0

we get λ^{∗} = 1, x_{1}^{∗} = x_{2}^{∗} = 0.5, i.e., the function is minimized at f(0.5, 0.5) = 0.5. We note that f(x_{1}, x_{2}) and g(x_{1}, x_{2}) have the same gradients at x_{1}^{∗} =

x_{2}^{∗} = 0.5:
▽f(0.5, 0.5) = ▽h(0.5, 0.5) = [1, 1]^{T}

• Consider these two problems:

\{ \begin{matrix}min: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≥ 0 \end{matrix}  \{ \begin{matrix}max: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≤ 0 \end{matrix}

These problems are the two middle cases (2 and 3) in Table 3.21. They have the same Lagrangian as above with the same results x_{1}^{∗} = x_{2}^{∗} = 0.5 and μ^{∗} = 1 > 0. According to Table 3.21, x^{∗} is the solution for the minimization problem subject to g(x) ≥ 0 (left), or the maximization problem subject to g(x) ≤ 0.

• Consider these two problems:

\{ \begin{matrix}max: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≥ 0 \end{matrix}  \{ \begin{matrix}min: & f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} \\ s. t.: & g(x_{1}, x_{2}) = x_{1} + x_{2} − 1 ≤ 0 \end{matrix}

These problems are cases 1 and 4 in Table 3.21. Again they have the same Lagrangian as in the previous cases and the same results x_{1}^{∗} = x_{2}^{∗} = 0.5, and μ^{∗} = 1 > 0. According to Table 3.21, x^{∗} is not the solution of either of the two problems, i.e., the constraint is inactive. We therefore need to assume μ^{∗} = 0 and solve the following equations for an unconstrained problem:

\frac{∂L}{∂x_{1}} = 2x_{1} = 0, \frac{∂L}{∂x_{2}} = 2x_{2} = 0

Now we get x_{1}^{∗} = x_{2}^{∗} = 0 and x^{∗} = 0, with minimum f(x^{∗}) = 0. This is the solution for the minimization problem (right), at which g(x^{∗}) ≠ 0, i.e., the constraint is inactive. However this is not the solution for the maximization problem as the function is unbounded from above.

3.3 Duality and KKT Conditions

Previously we considered the constrained minimization problem:

\{ \begin{matrix}min: & f_{p}(x) \\ s. t.: & h(x) = 0,  g(x) ≤ 0 or g(x) ≥ 0 \end{matrix}  (3.37)

with its Lagrangian:

L(x, λ, μ) = f_{p}(x) − λ^{T} h(x) − μ^{T} g(x)  (3.38)

The optimal solution x^{∗} can be found by solving the following equation systems:

▽_{x}L(x, λ, μ) = 0, ▽_{λ}L(x, λ, μ) = 0, ▽_{μ}L(x, λ, μ) = 0  (3.39)

at which the objective function reaches its minimum p^{∗} = f_{p}(x^{∗}).

In particular, if the objective function f_{p}(x) and all inequality constraints g_{i}(x) are convex and the equality constraints h_{i}(x) are linear (affine), i.e., the feasible region is convex (Section A.8.3), then the problem is a convex optimization with some nice properties as shown below.

This constrained minimization problem given above can be called the primal problem, as there also exists an equivalent dual problem, which may be easier to solve. To construct the dual problem, we first define the dual function as the lower bound of the Lagrangian in Eq. (3.38) to be minimized:

f_{d}(λ, μ) = min_{x} L(x, λ, μ) = min_{x} [f_{p}(x) − λ^{T} h(x) − μ^{T} g(x)]  (3.40)

We further find the tightest lower bound, the maximum of the dual function f_{d}(λ, μ) over its variables λ and μ, by solving the following dual problem:

\{ \begin{matrix}max: & f_{d}(λ, μ) = min_{x} L(x, λ, μ) \\ s. t.: & μ ≤ 0 or μ ≥ 0 \end{matrix}  (3.41)

Here λ for the equality constrains of the primal problem can be either positive or negative, i.e., it is not constrained in the dual problem. However, the sign of μ for the inequality constraints of the primal minimization problem needs to satisfy the following according to Eq. (3.22) (the second inequality for this minimization primal problem):

μ^{T} g(x) ≥ 0  (3.42)

i.e., μ is subject to the inequality constraint of the dual problem. We denote the optimal solution of the dual problem by {λ^{∗}, μ^{∗}}, and the maximum of the dual objective function by d^{∗} = f_{d}(λ^{∗}, μ^{∗}).

Obviously the primal objective function of any feasible x is always no less than its lower bound, the dual objective function of any feasible {λ, μ}:

f_{p}(x) ≥ f_{d}(λ, μ)  (3.43)

and this inequality also holds for the optimal solutions x^{∗} and {λ^{∗}, μ^{∗}}:

p^{∗} = f_{p}(x) ≥ f_{d}(λ, μ) = d^{∗}  (3.44)

The difference p^{∗} −d^{∗} is defined as the duality gap. Obviously p^{∗} −d^{∗} ≥ 0 always holds and the duality is called weak duality. In some problems where certain conditions (Slater’s conditions) are satisfied, p^{∗} = d^{∗}, i.e., d^{∗} is the tightest lower bound, the reality is called strong duality.

We now consider the necessary and sufficient conditions for the solutions of both the primal and dual problems. We first introduce the Karush-Kuhn-Tucker (KKT) conditions:

• Stationarity: x^{∗} maximizes or minimizes the Lagrangian, i.e.,

▽_{x}L(x^{∗}, λ, μ) = 0  (3.45)

• Primal feasibility: x^{∗} must be inside the feasible region:

h_{i}(x^{∗}) = 0, g_{j}(x^{∗}) ≤ 0 or g_{j}(x^{∗}) ≥ 0, (i = 1, ⋯, m, j = 1, ⋯, n) (3.46)

• Dual feasibility: All multipliers in μ for the inequalities must take proper sign as specified by Table 3.21:

μ_{j}^{∗} ≤ 0 or μ_{j}^{∗} ≥ 0, (j = 1, ⋯, n)  (3.47)

• Complementarity: Either μ_{j}^{∗} or g_{j}(x^{∗}) must be zero depending on whether the jth constraint is inactive or not:

– if inactive, then g_{j}(x^{∗}) ≠ 0, but μ_{j}^{∗} = 0;

– if active, then μ_{j}^{∗} ≠ 0, but g_{j}(x^{∗}) = 0. Combining both cases, we get (Eq. (3.25)):

(μ^{∗})^{T} g(x^{∗}) = 0  (3.48)

We now prove the following theorem stating that the KKT conditions and both necessary and sufficient for the solutions of the primal and dual problems to exist.

Theorem:

• (Necessary) If x^{∗} and {λ^{∗}, μ^{∗}} are solutions for the respective primal and dual problems with strong duality, then they satisfy all KKT conditions.

• (Sufficient) If x^{∗} and {λ^{∗}, μ^{∗}} satisfy all KKT conditions, then they are solutions for the respective primal and dual problems, and strong duality holds, when the primal objective function f_{p}(x^{∗}) and the inequality constraints g_{j}(x) are convex, and equality constraints h_{j}(x) are linear.

Proof:

• Necessary conditions:

As x^{∗} and {λ^{∗}, μ^{∗}} are respectively the primal and dual solutions, the primal and dual feasibilities must be satisfied, and so is stationarity as the optimal solution x^{∗} satisfies Eq. (3.39). In addition, we also have h(x^{∗}) = 0. To show complementarity also holds, consider

d^{∗} = f_{d}(λ^{∗}, μ^{∗}) = min_{x} L(x, λ^{∗}, μ^{∗})
= min_{x} [f_{p}(x) − (λ^{∗})^{T} h(x) − (μ^{∗})^{T} g(x)]
≤ f_{p}(x^{∗}) − (λ^{∗})^{T} h(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
= f_{p}(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
≤ f_{p}(x^{∗}) = p^{∗}  (3.49)

The last inequality is due to Eq. (3.42). But as d^{∗} = p^{∗} due to the strong duality, the inequalities need to be equalities, i.e., (μ^{∗})^{T} g(x^{∗}) = 0 and complementarity holds.

• Sufficient conditions:

If x^{∗} and {λ^{∗}, μ^{∗}} satisfy the KKT conditions, then they are feasible for the respective primal and dual problems. Also, as f_{p}(x) and all g_{j}(x) are convex and h_{j}x) are linear, Lagrangian in Eq. (3.38) as a sum of convex functions is also convex (see Section A.8.3), and stationarity in Eq. (3.45) indicates x^{∗} is a global minimum, therefore

d^{∗} = f_{d}(λ^{∗}, μ^{∗}) = L(x^{∗}, λ^{∗}, μ^{∗}) = f_{p}(x^{∗}) − (λ^{∗})^{T} h(x^{∗}) − (μ^{∗})^{T} g(x^{∗})
= f_{p}(x^{∗}) = p^{∗}  (3.50)

where the second line is due to the KKT conditions in Eqs. (3.46) and (3.48).

In summary, we see that the KKT conditions are necessary and satisfied by all solutions of constrained optimization problems, and they can also be sufficient for the solution of such problems if the primal problem further satisfies some additional conditions of convexity, such as in quadratic program if the quadratic objective function is also positive definite. In such a case, strong duality holds and the solution of the dual problem is the tightest lower bound of the primal, the same as the solution of the primal.

Consider, as an example of the inequality constrained minimization problems, the support vector machine for binary classification (Chapter 14). To find the decision boundary that optimally separates two classes, we need to solve the following minimization problem constrained by a set of inequalities imposed by the data samples {(x_{n}, y_{n}), n = 1, ⋯, N} in the training set:

\{ \begin{matrix}min: & \frac{1}{2}||w||^{2} \\ s. t.: & y_{n}(x_{n}^{T} w + b) ≥ 1 (n = 1, ⋯, N) \end{matrix}  (3.51)

where w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}. We note that the quadratic objective function is convex and so are the affine inequality constraints, i.e., this is a convex problem. The solution w with minimum norm ||w||^{2} subject to the inequality constraints can be found by solving the dual problem in terms of the Lagrange multipliers in α = [α_{1}, ⋯, α_{N}]^{T}, subject to certain inequality constraints (detailed derivation in Section 14):

\{ \begin{matrix}max: & ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m}) \\ s. t.: & ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N) \end{matrix}  (3.52)

As the dual function can be written a quadratic form:

α^{T} 1 − \frac{1}{2}α^{T} Qα  (3.53)

where Q is an N × N symmetric matrix of which the component in the mth row and nth column is Q(m, n) = y_{m}y_{n}x_{m}^{T}x_{n}, this is a quadratic programming problem, which can be solved by the interior point algorithm to be discussed in Section 3.7.

3.4 Linear Programming (LP)

The basic problem in linear programming (LP) is to minimize/maximize a linear objective function f(x_{1}, ⋯, x_{N}) = ∑_{i=1}^{N} c_{i}x_{i} of N variables x_{1}, ⋯, x_{N} under a set of M linear constraints:

\begin{matrix}max: & f(x_{1}, ⋯, x_{N}) = ∑_{i=1}^{N} c_{i}x_{i} \\ s. t.: & \{ \begin{matrix}∑_{i=1}^{N} a_{1i}x_{i} ≤ b_{1} \\ ⋯ & ⋯ & ⋯ & ⋯ \\ ∑_{i=1}^{N} a_{Mi}x_{i} ≤ b_{M} \\ x_{1} ≥ 0, ⋯, x_{N} ≥ 0 \end{matrix} \end{matrix}  (3.54)

The LP problem can be more concisely represented in matrix form:

max: c^{T} x
s. t.: \{ \begin{matrix}Ax − b ≤ 0 \\ x ≥ 0 \end{matrix} (3.55)

where

x = [ x_{1} ; ⋮ ; x_{N} ] c = [ c_{1} ; ⋮ ; c_{N} ] b = [ b_{1} ; ⋮ ; b_{M} ] A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ]

(3.56)

and the smaller or greater than sign in a vector inequality should be understood as an element-wise relationship between the corresponding elements of the vectors on both sides.

For example, the objective function could be the total profit, the constraints could be some limited resources. Specifically, x_{1}, ⋯, x_{N} may represent the quantities of N different types of products, c_{1}, ⋯, c_{N} represent their unit prices, and a_{ij} represent the consumption of the ith resource by the jth product, the goal is to maximize the profit f(x_{1}, ⋯, x_{N}) subject to the constraints imposed by the limited resources b_{1}, ⋯, b_{M}.

Here all variables are assumed to be non-negative. If there exists a variable that is not restricted, it can be eliminated. Specifically, we can solve one of the constraining equations for the variable and use the resulting expression of the variable to replace all its appearances in the problem. For example:

\begin{matrix}max: & f(x_{1}, x_{2}, x_{3}) = 2x_{1} − x_{2} + 3x_{3} \\ s. t.: & \{ \begin{matrix}x_{1} − 2x_{2} + x_{3} = 3 \\ 3x_{1} − x_{2} + 4x_{3} = 10 \\ x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix} \end{matrix}  (3.57)

Solving the first constraining equation for the free variable x_{1}, we get x_{1} = 3+2x_{2}−x_{3}. Substituting this into the objective function and the other constraint, we get 2x_{1} − x_{2} + x_{3} = 3x_{2} + x_{3} + 6 and 5x_{2} + x_{3} = 1, the problem can be reformulated as:

\begin{matrix}max: & f(x_{2}, x_{3}) = 3x_{2} + x_{3} + 6 \\ s. t.: & \{ \begin{matrix}5x_{2} + x_{3} = 1 \\ x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix} \end{matrix}  (3.58)

Given the primal LP problem above, we can further find its dual problem. We first construct the Lagrangian of primal problem:

L(x) = c^{T} x − y^{T} (Ax − b) = (c − A^{T} y)^{T} x + y^{T} b  (3.59)

where y = [y_{1}, ⋯, y_{M}]^{T} contains the Lagrange multipliers for the inequality constraints Ax − b ≤ 0, and y ≥ 0 according to Table 3.21. We define the dual function as the maximum of L(x) over x:

f_{d}(y) = max_{x} L(x) = max_{x}[c^{T} x−y^{T} (Ax−b)] = max_{x}[(c−A^{T} y)^{T} x+y^{T} b] (3.60)

To find the value of x that maximizes L(x), we set its gradient to zero and get:

▽_{x}L(x) = ▽_{x}[(c − A^{T} y)^{T} x + y^{T} b] = c − A^{T} y = 0  (3.61)

Substituting this into L_{d}(y) = max_{x} L(x) we get: f_{d}(y) = y^{T} b, which is the upper bound of f_{p}(x), under the constraint that c − A^{T} y ≤ 0 so that (c − A^{T} y)^{T} x ≤ 0 contributes negatively to L(x) (as x ≥ 0):

f_{d}(y) = b^{T} y ≥ L(x) = c^{T} x − y^{T} (Ax − b) ≥ c^{T} x = f_{p}(x)  (3.62)

This upper bound needs to be minimized with respect to y, to become the tightest upper bound, i.e., the original primal problem of maximization in Eq. (3.55) has now been converted into the following dual problem of minimization:

\{ \begin{matrix}max : & c^{T} x \\ s.t. & Ax − b ≤ 0, x ≥ 0 \end{matrix} ⟺ \{ \begin{matrix}min & b^{T} y \\ s.t. & A^{T} y − c ≥ 0, y ≥ 0 \end{matrix} (3.63)

Following the same approach, we can also find the dual of a minimization primal problem:

\{ \begin{matrix}min : & c^{T} x \\ s.t. & Ax − b ≤ 0, x ≥ 0 \end{matrix} ⟺ \{ \begin{matrix}max & b^{T} y \\ s.t. & A^{T} y − c ≤ 0, y ≤ 0 \end{matrix}
⟺ \{ \begin{matrix}max & −b^{T} y \\ s.t. & A^{T} y + c ≥ 0, y ≥ 0 \end{matrix} (3.64)

Moreover, we can show that the dual of a dual problem is the original primal problem.

If either the primal or the dual is feasible and bounded, so is the other, and they form a strong duality, the solution d^{∗} of the dual problem is the same as the solution p^{∗} of the primal problem. Also, as the primal and dual problems are completely symmetric.

An inequality constrained LP problem can be converted to an equality constrained problem by introducing slack variables:

∑_{i=1}^{N} a_{i}x_{i} ≤ b ⟹ ∑_{i=1}^{N} a_{i}x_{i} + s = b, (s ≥ 0)  (3.65)

so that it can be reformulated to take the standard form:

max: z = f(x) = c^{T} x = ∑_{j=1}^{N} c_{j}x_{j}

s. t.: \{ \begin{matrix}∑_{i=1}^{N} a_{1i}x_{i} & +s_{1} & & & & = b_{1} \\ ∑_{i=1}^{N} a_{2i}x_{i} & & +s_{2} & & & = b_{2} \\ ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ \\ ∑_{i=1}^{N} a_{Mi}x_{i} & & & & +s_{M} & = b_{M} \\ x_{1} ≥ 0, & ⋯, & x_{N} ≥ 0, & s_{1} ≥ 0, & ⋯ & s_{M} ≥ 0 \end{matrix} (3.66)

We further redefine the following:

• x is an N + M dimensional augmented variable vector that includes the M slack variables {s_{1}, ⋯, s_{M}} as well as the N original variables {x_{1}, ⋯, x_{N}}:

x = [x_{1}, x_{2}, ⋯, x_{N}, s_{1}, s_{2}, ⋯, s_{M}]^{T}  (3.67)

• A is redefined as an M × (N + M) augmented coefficient matrix that includes coefficients for both types of variables:

A = [ \begin{matrix}a_{11} & a_{12} & ⋯ & a_{1N} & 1 & 0 & ⋯ & 0 \\ a_{21} & a_{22} & ⋯ & a_{1N} & 0 & 1 & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ & ⋮ & ⋮ & ... & ⋮ \\ a_{M1} & a_{M2} & ⋯ & a_{MN} & 0 & ⋯ & 0 & 1 \end{matrix} ] = [ A_{M×N} | I_{M×M} ]

(3.68)

where A_{M×N} are the coefficients for the N original variables {x_{1}, ⋯, x_{N}}, and the identity matrix I_{M×M} = [e_{1}, ⋯, e_{M}] is for the unity coefficients of the M slack variables {s_{1}, ⋯, s_{M}}, each of which appears in the M constraints only once.

Now the LP problem can be expressed in the standard form (original form on the left, with x and A redefined on the right):

max: c^{T} x s. t.: \{ \begin{matrix}Ax + s = b \\ x ≥ 0, s ≥ 0 \end{matrix}  or  max: c^{T} x s. t.: \{ \begin{matrix}Ax = b \\ x ≥ 0 \end{matrix}  (3.69)

An LP problem can be viewed geometrically. If normalize c so that ||c|| = 1, then the objective function f(x) = c^{T} x becomes the projection of x onto c. Each of the M equality constraints in Ax = b represents a hyper-plane in the N-D vector space perpendicular to its normal direction a_{j} = [a_{i1}, ⋯, a_{IN}]^{T}:

∑_{j=1}^{N} a_{ij}x_{j} = a_{j}^{T} x = b_{i},  (i = 1, ⋯, M)  (3.70)

Figure 3.6 Fundamental Theorem of Linear Programming

Also, each of the N non-negativity conditions x_{j} ≥ 0 in x ≥ 0 corresponds to a hyper-plane perpendicular to the jth standard basis vector e_{j}. In general, in an N-D space, if none of the hyper-planes is parallel to any others, then no more than N hyper-planes intersect at one point. For example, in a 2-D or 3-D space, two straight lines or three surfaces intersect at a point. Here for the LP problem in the N-D space, the total number of intersection points formed by the N + M hyper-planes is

C_{M+N}^{N} = C_{M+N}^{M} = \frac{(M + N)!}{N! M!}  (3.71)

The polytope enclosed by these N + M hyper-planes is called the feasible region, in which the optimal solution must lie. The vertices of the polytopic feasible region are a subset of the C_{M+N}^{N} intersections.

Fundamental theorem of linear programming

The optimal solution x^{∗} of a linear programming problem formulated above is either a vertex of the polytopic feasible region P, or lies on a hyper-surface of the polytope, on which all points are optimal solutions.

Proof:

Assume the optimal solution x^{∗} ∈ P is interior to the polytopic feasible region P (as shown in Fig. 3.6), then there must exist some ϵ > 0 such that the hypersphere of radius ϵ centered at x^{∗} is inside P. Evaluate the objective function at a point on the hyper-sphere

x′ = x^{∗} + ϵ c/||c|| ∈ P  (3.72)

we get

f(x′) = c^{T} x′ = c^{T} (x^{∗} + ϵ c/||c||)
= c^{T} x^{∗} + ϵ c^{T} c/||c|| = c^{T} x^{∗} + ϵ ||c|| > c^{T} x^{∗} = f(x^{∗}) (3.73)

i.e., x^{∗} cannot be the optimal solution as assumed. This contradiction indicates that an optimal solution x^{∗} must be on the surface of P, either at one of its vertices or on one of its surfaces. Q.E.D.

Based on this theorem, the optimization of a linear programming problem could be solved by exhaustively checking each of the C_{N+M}^{N} intersections formed by N of the N + M hyper-planes to find (a) whether it is feasible (satisfying all N + M constraints), and, if so, (b) whether the objective function f(x) is maximized at the point. Specifically, we choose N of the N +M equations for the the constraints and solve this N-equation and N-unknown linear system to get the intersection point of N corresponding hyper-planes. This brute-force method is most straight forward, but the computational complexity is high when N and M, and therefore C_{N+M}^{N}, become large.

Example 3.2

max: f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2}_{s. t.: \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix}}

or

max: f_{p}(x) = c^{T} x
s. t.: \{ \begin{matrix}Ax − b ≤ 0 \\ x ≥ 0 \end{matrix}

where

x = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , c = [ \begin{matrix}2 \\ 3 \end{matrix} ] , b = [ \begin{matrix}18 \\ 60 \\ 40 \end{matrix} ] , A = [ \begin{matrix}2 & 1 \\ 6 & 5 \\ 2 & 5 \end{matrix} ]

This problem has N = 2 variables with the same number of non-negativity constraints and M = 3 linear inequality constraints, as shown in Fig. 3.7. The n + m = 5 straight lines form C_{N+M}^{N} = C_{5}^{2} = 10 intersections, out of which 5 are the vertices of the polygonal feasible region satisfying all constraints. The value of the objective function f_{p}(x_{1}, x_{2}) = c^{T} x = c_{1}x_{1} + c_{2}x_{2} is proportional to the projection of the 2-D variable vector x = [x_{1}, x_{2}]^{T} onto the coefficient vector c = [2, 3]^{T}. The goal of this linear programming problem is to find a point x inside the polygonal feasible region with maximum f(x) = c^{T} x (the projection of x onto vector c if ||c|| = 1). Geometrically, this can be understood as to push the hyper-plane c^{T} x = z, here a line in 2-D space, its normal direction c, away from the origin along to eventually arrive at the vertex of the polytopic feasible region farthest away from the origin.

This LP problem can be converted into the standard form:

\begin{matrix}max: & f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix} \end{matrix}  ⟹  \begin{matrix}max: & f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} + s_{1} = 18 \\ 6x_{1} + 5x_{2} + s_{2} = 60 \\ 2x_{1} + 5x_{2} + s_{3} = 40 \\ x_{1} ≥ 0, x_{2} ≥ 0, s_{1} ≥ 0, s_{2} ≥ 0, s_{3} ≥ 0 \end{matrix} \end{matrix}

Figure 3.7 Simplex Algorithm Example (Original Problem)

Listed in the table below are the C_{M+N}^{N} = C_{5}^{2} = 10 intersections x, called basic solutions, together with the corresponding slack variables s_{1}, s_{2}, s_{3} and the objective function value f(x) (proportional to the projection of x onto c). We see that out of the ten basic solutions, five are feasible (vertices of the polytopic feasible region) with all M + N = 5 variables taking non-negative values, while the remaining five are not feasible, as some of the slack variables are negative, and thereby violating the constraints. Out of the five feasible solutions, the one at x^{∗} = (5, 6) is optimal with maximum objective function value f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} = 2 × 5 + 3 × 6 = 28.

x_{1} x_{2} s_{1} s_{2} s_{3}  feasibility c^{T} x/||c|| = 1
1 5 6 2 0 0 feasible 28/7.77
2 6.25 5.5 0 −5 0 infeasible  29/8.04
3 7.5 3 0 0 10 feasible 24/6.66
4 20 0 −2 −60 20 infeasible  40/11.1
5 10 0 −2 0  20 infeasible  20/5.55
6  9  0 0  6  22  feasible  18/4.10
7  0  8 10 20  0  feasible  24/6.66
8  0 12 6  0 −20 infeasible  36/9.98
9  0 18 0 −30 −50 infeasible  54/14.99
10 0  0 18 60 40  feasible  0/0.00

This primal maximization problem can be converted into the dual minimization problem:

min: f_{d}(y_{1}, y_{2}, y_{3}) = 18y_{1} + 60y_{2} + 40y_{3}_{s. t.: \{ \begin{matrix}2y_{1} + 6y_{2} + 2y_{3} ≥ 2 \\ y_{1} + 5y_{2} + 5y_{3} ≥ 3 \\ y_{1} ≥ 0, y_{2} ≥ 0, y_{3} ≥ 0 \end{matrix}}

or

min: f_{d}(y) = b^{T} y
s. t.: \{ \begin{matrix}A^{T} y − c ≥ 0 \\ y ≥ 0 \end{matrix}

As shown in Fig. 3.8, the five planes, including m = 2 from the constraining inequalities and n = 3 for the non-negative constraints y_{1} ≤ 0, y_{2} ≤ 0, and y_{3} ≤ 0 for the three variables, form 10 intersection points:

(y_{1}, y_{2}, y_{3}) feasibility objective function
1  (1, 0, 0)  infeasible  18
2  (3, 0, 0)  feasible  54
3 (0, 1/3, 0) infeasible  20
4 (0, 3/5, 0)  feasible  36
5  (0, 0, 1)  feasible  40
6 (0, 0, 3/5) infeasible  24
7 (0, 1/5, 2/5) feasible  28
8 (1/2, 0, 1/2) feasible  29
9 (−2, 1, 0) infeasible  24
10  (0, 0, 0)  infeasible  0

We see that at the feasible point (0, 1/5, 2/5), the dual function f_{d}(y_{1}, y_{2}, y_{3}) = 18y_{1} +60y_{2} +40y_{3} is minimized to be 28, the same as the maximum of the primal function f_{p}(x_{1}, x_{2}) = 2x_{1} + 3x_{2} at (5, 6).

3.5 The Simplex Algorithm

The simplex algorithm finds the optimal solution of a LP problem by an iterative process that traverses along a sequence of edges of the polytopic feasible region,

Figure 3.8 Simplex Algorithm Example (Dual Problem)

starting at the origin and through a sequence of vertices x with progressively greater objective value f(x), until eventually reaching the optimal solution. By doing so, it avoids checking exhaustively all vertices of the feasible region for optimality.

We first consider the equality constraint Ax = [A_{M×N} | I_{M×M}]x = b of the standard LP problem. This is an under-determined linear system of N + M variables but only M equations, only M of its N + M columns of the coefficient matrix A are independent (assuming rank(A) = M). Initially we choose the M columns of the identity coefficient matrix I as the independent columns, but subsequently we can choose any other M columns of A as the independent columns and convert them into a standard basis vector by Gauss-Jordan elimination together with the corresponding variables in x. The resulting equation Ax = b remains equivalent to the original one, i.e., the M equality constraints are always preserved and never violated in the process.

For convenience of discussion and without loss of generality, we could reorder (not actually carried out) the N + M columns in A, together with the corresponding variables in x, so that the constraining equation would always takes the following form:

Ax = [A_{n} I] [ \begin{matrix}x_{n} \\ x_{b} \end{matrix} ] = A_{n}x_{n} + I x_{b} = b  (3.74)

Now the variables in the M-D vector x_{b} corresponding to the M independent columns in I are the basic variables, while the N-D vector x_{n} corresponding to the remaining N dependent columns, now denoted by A_{n}, are the non-basic variables. This equation will always hold if x_{n} = 0 and x_{b} = b. Such a solution is called a basic solution of the linear system Ax = b:

x = [ \begin{matrix}x_{n} \\ x_{b} \end{matrix} ] = [ \begin{matrix}0 \\ b \end{matrix} ]  (3.75)

Initially, A_{n} is the coefficient matrix in the inequality constraint Ax ≤ b, and the corresponding non-basic variables x_{n} = 0 is actually the origin. But through the iteration, we will keep converting some other columns of A into standard basis vectors {e_{1}, ⋯, e_{M}}, the column vectors of I, by Gauss-Jordan elimination. The variables in x corresponding to the new standard basis vectors become the basic variables, while those corresponding to columns that become non-standard basis vectors become non-basis variables.

The basic solutions corresponding the C_{N+M}^{N} ways to choose M of the N + M columns of A as independent are actually the intersections formed by any N of the N + M constraining hyper-planes in the N-D space. As noted before, only a subset of these C_{N+M}^{N} basic solutions satisfy the M constraints, and they are called basic feasible solutions (BFS). The goal of the iterative process of the simplex algorithm is to find the optimal basic feasible solution that maximizes f(x) without exhausting all C_{N+M}^{N} possibilities. This is done by selecting the columns in such a way that the value of the objective function will always be maximally increased, until eventually we find the optimal solution x^{∗} at one of the vertex of the polytopic feasible region.

Specially, the implementation of the simplex algorithm is based on a tableau with N + M + 1 columns and M + 1 rows, initialized as below:

• The first M rows are for the coefficients in the equality constraints ∑_{j=1}^{N} a_{ij}x_{j}+ s_{i} = b_{i} (i = 1, ⋯, M). An additional (M + 1)st row at the bottom is for the coefficients in the objective function f(x) = ∑_{j=1}^{N} c_{j}x_{j}, which is zero initially at the origin x_{1} = ⋯ = x_{N} = 0, i.e., z − c_{1}x_{1} − ⋯ − c_{N}x_{N} = 0.

• The first N + M columns contain the coefficients for the N original variables {x_{1}, ⋯, x_{N}} and the M slack variables {s_{1}, ⋯, s_{M}}, and an additional (N + M + 1)st column is for the constants b = [b_{1}, ⋯, b_{M}]^{T} on the righthand side of the M constraint equations. The last element of the column is the right-hand side of the objective equation z = f(x), which is initially zero.

basic variables x_{1} x_{2}

⋯

x_{N} s_{1} s_{2}

⋯

s_{M}

basic solution

s_{1}  a_{11} a_{12} ⋯ a_{1N} 1 0 ⋯ 0  b_{1}
s_{2}  a_{21} a_{22} ⋯ a_{2N} 0 1 ⋯ 0  b_{2}
⋮  ⋮  ⋮ ...  ⋮  ⋮ ⋮ ... ⋮  ⋮
s_{M}  a_{M1} a_{M2} ⋯ a_{MN} 0 0 ⋯ 1  b_{M}
z  −c_{1} −c_{2} ⋯ −c_{M} 0 0 ⋯ 0  0

(3.76)

At this initial stage of the iteration, the N original variables x_{n} = [x_{1}, ⋯, x_{N}]^{T} are the non-basic variables, while the M slack variables x_{b} = [s_{1}, ⋯, s_{M}]^{T} are the basic variables and their coefficients form an M × M identity matrix I, composed of the standard basis vectors. The corresponding feasible basic solution is simply

x_{b} = [s_{1}, ⋯, s_{M}]^{T} = b, x_{n} = [x_{1}, ⋯, x_{N}]^{T} = 0  (3.77)

that satisfies all the constraints Ax = A_{n}x_{n} + Ix_{b} = b.

In each of the subsequent iterations, we will select one of the non-basic variables, called the entering variable, to replace one of the basic variables, called the leaving variable, in such a way that the value of the objective function value z = f(x) in the last row will be maximally increased, while all constraints remain satisfied. Here are the steps in each iteration:

• Selection of the entering variable

This section is based on the maximization of z = f(x). We select x_{j} in the jth column of the tableau if −c_{j} is most negative, i.e., x_{j} is most heavily weighted by c_{j} = max{c_{1}, ⋯, c_{N}}, so that it will increase z = ∑_{j=1}^{N} c_{j}x_{j} more than any other x_{k} (k ≠ j).

• Selection of the leaving variable

This selection is based on the constraints imposed on the selected entering variable x_{j}. In general, the restriction on x_{j} set by the kth constraint ∑_{j=1}^{N} a_{kj}x_{j} ≤ b_{k} is x_{j} ≤ b_{k}/a_{kj} when a_{kj} > 0. If b_{i}/a_{ij} = min{b_{k}/a_{1j}, ⋯, b_{k}/a_{Mj}}, then the ith constraint is most restrictive on x_{j}, we will therefore select the corresponding basic variable s_{i} as the leaving variable, i.e., it becomes a non-basic variable to be set to zero, so that x_{j} can be maximally increased without violating the constraints. If all a_{kj} < 0 for all k = 1, ⋯, M, variable x_{j} is not bounded.

• Gauss-Jordan elimination based on Pivoting

To convert the entering variable x_{j} to a basic variable to replace the leaving variable s_{i}, we need to turn the corresponding jth column into a standard basis vector e_{i}. This is realized by pivoting on a_{ij} in the following steps: 1.Divide the pivot row r_{i} by a_{ij}: r_{i} ← r_{i}/a_{ij}. Now a_{ij} = 1. 2.Subtract a_{kj}r_{i} from the kth row: r_{k} ← r_{k} − a_{kj}r_{i} so that a_{kj} = 0 for all k = 1, ⋯, M + 1, k ≠ i, including the last row r_{M+1}. Now the jth column corresponding to the entering variable x_{j} becomes a standard basis vector e_{i}, i.e., x_{j} becomes a basic variable that takes the value of b_{i}, and the column corresponding to the leaving variable s_{i} is no longer a standard basis vector, and s_{i} = 0 as it is now a non-basic variable.

As Gauss-Jordan elimination converts the linear equations to a set of equivalent equations, the constraints remain satisfied through out the process. Although the membership of the basic and non-basic variable groups keeps changing in the iteration, so long as x_{b} = b and x_{n} = 0, the M constraint equations Ax = A_{n}x_{n} + Ix_{b} = b always holds.

This iterative process keeps replacing the slack variables {s_{1}, ⋯, s_{M}} (the basic variables initially) by the original variables {x_{1}, ⋯, x_{N}} (the non-basic variables initially), one at a time, until all original variables become basic variables and all elements in the last row are non-negative.

The final result can be read out directly from the tableau. The variables corresponding to the M standard basis vectors e_{i} (i = 1, ⋯, M) are the final basic variables that take the values in b in the right-most column of the tableau. They form the optimal basic solution, with the maximum of the objective function z = f(x) given by the last element also in the last column. The remaining n variables corresponding to non-standard columns are non-basic variables that take the value zero. When M > N (more constraints than variables), some of the slack variables may remain in the basic variable group taking non-zero values; when M < N, some of the original variable may be in the non-basic group taking the value zero.

Example 3.3 We re-solve the LP problem considered in the previous examples, now in the standard form:

\begin{matrix}max: & f(x) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} + s_{1} = 18 \\ 6x_{1} + 5x_{2} + s_{2} = 60 \\ 2x_{1} + 5x_{2} + s_{3} = 40 \\ x_{1} ≥ 0, x_{2} ≥ 0, s_{1} ≥ 0, s_{2} ≥ 0, s_{3} ≥ 0 \end{matrix} \end{matrix}

The standard form is further converted to a tableau as shown below. The leftmost column indicates the basic variables, the next N = 2 columns are for the variables x and y,the next M = 3 columns are for the slack variables u, v, and w, the right most column is for the constants b_{1}, b_{2}, and b_{3} on the right-hand side of the equations.

x_{1} x_{2} s_{1} s_{2} s_{3} b
s_{1} 2 1 1 0 0 18
s_{2} 6 5 0 1 0 60
s_{3} 2 5 0 0 1 40
z −2 −3 0 0 0 0

In this initial state, the basic variables are s_{1} = 18, s_{2} = 60, and s_{3} = 40, the non-basic variables are x_{1} = x_{2} = 0. The corresponding basic feasible solution is at the origin.

• Select column j = 2 as the pivot column, as −c_{2} = −3 < −c_{1} is the most negative coefficient.

• Select r_{3} as the pivot row, as b_{3}/a_{32} = 40/5 = 8 = min{18/1 = 18, 60/5 = 12, 40/5 = 8}.

• Divide pivot row r_{i} = r_{3} = [2, 5, 0, 0, 1, 40] by the pivot element a_{ij} = 32 = 5 to get r_{i} = r_{3} = [0.4, 1, 0, 0, 0.2, 8].

• Subtract a_{kj}r_{k} = a_{k2}r_{k} from row r_{k}, so that a_{kj} = a_{k2} = 0, (k = 1, ⋯, M + 1 = 4, k ≠ i), including the last row r_{4}.

x_{1} x_{2} s_{1} s_{2} s_{3} b
s_{1} 1.6 0 1 0 −0.2 10
s_{2} 4  0 0 1 −1 20
x_{2} 0.4 1.0 0 0 0.2 8
z −0.8 0 0 0 0.6 24

The entering variable x_{j} = x_{2} becomes a basic variable, and the leaving variable s_{i} = s_{3} becomes a non-basic variable, replaced by new basic variable x_{2}. The corresponding basic feasible solution is at x_{1} = 0, x_{2} = b_{3} = 8, with s_{1} = 10, s_{2} = 20, s_{3} = 0. The objective function value is z = c^{T} x = 24.

• Select column j = 1 as the pivot column, as −c_{1} = −0.8 is the most (only) negative coefficient in the last row.

• Select row i = 2 as the pivot row, as the ratio b_{2}/a_{21} = 20/4 = 5 = min{10/1.6 = 6.25, 20/4 = 5, 8/0.4 = 20}.

• Divide pivot row r_{i} = r_{2} = [4, 0, 0, 1, −1, 20] by the pivot element a_{ij} = a_{21} = 4 to get r_{i} = r_{2} = [1, 0, 0, 0.25, −0.25, 5].

• Subtract a_{kj}r_{k} = a_{k1}r_{k} from row r_{k}, so that a_{kj} = a_{k1} = 0, (k = 1, ⋯, M + 1 = 4), including the last row r_{4}.

x_{1} x_{2} s_{1} s_{2}  s_{3}  b
s_{1} 0 0 1 −0.4 0.2 2
x_{1} 1 0 0 0.25 −0.25 5
x_{2} 0 1 0 −0, 1 0.3 6
z 0 0 0 0.2  0.4 28

The entering variable x_{1} becomes a basic variable, and the leaving variable s_{1} becomes a non-basic variable, replaced by the new basic variable x_{1}. The corresponding basic feasible solution is at x_{1} = b_{2} = 5, x_{2} = b_{3} = 6, with s_{1} = 2, s_{2} = s_{3} = 0. The objective function value is z = c^{T} x = 28. Now that both x_{1} and x_{2} have become basic variables, the optimal solution has been obtained.

This problem has C_{N+M}^{N} = C_{5}^{2} = 10 basic solutions, corresponding to the same number of intersections, out of which five are feasible, as previously obtained. The simplex method finds three of these feasible solutions, starting from x_{1} = x_{2} = 0 at the origin with z = 0, through the vertex at x_{1} = 0 and x_{2} = 8 with z = 24, to the optimal solution at x_{1} = 5 and x_{2} = 6 with z = 28.

The Matlab code for implementing the simplex algorithm is listed below. The function takes input including matrix A as well as the vectors b and c, and generates the optimal solution x^{∗}, the corresponding maximum value of the objective function f(x) = c^{T} x^{∗}, together with the values of the slack variable s.

function [x s z] = MySimplex(A,b,c)
[m n]=size(A)          % coefficient matrix for variables x
x=zeros(n,1);          % initial zero values for x
s=zeros(m,1);
A=[A eye(m) b; -c zeros(1,m+1)]; % initialization of tableau
[cmin, pc]=min( A(m+1,1:n+m) );  % pc, index of first pivot column
it=0;
while cmin<0           % iteration until all c's in last row are zero
it=it+1;
rmin=9e9;
for i=1:m          % find pivot row
    if A(i,n+m+1)~=0 & A(i,pc)>0
        w=A(i,n+m+1)./A(i,pc);
        if w<rmin
            pr=i;  % pr, index of the pivot row
            rmin=w;
        end
    end
end
p=A(pr,pc);        % get the pivot element
A(pr,:)=A(pr,:)/p; % modify pivot row
for i=1:m+1        % modify all k+1 rows
    if i~=pr
        A(i,:)=A(i,:)-A(pr,:)*A(i,pc);
    end
end
z=A(m+1,n+m+1);    % maximum value of objective function so far
[cmin, pc]=min( A(m+1,1:n+m) ); % pivot column of next iteration
end
end

3.6 Quadratic Programming (QP)

A quadratic programming (QP) problem is to minimize a quadratic function subject to some equality and/or inequality constraints:

min: f(x) = \frac{1}{2}[x − m]^{T} Q[x − m] = \frac{1}{2}x^{T} Qx + c^{T} x + c
s. t.: Ax ≤ b

(3.78)

where Q = Q^{T} is an N × N positive definite matrix, A an M × N matrix, and b is an M-D vector. Also, c = −Qm, c = m^{T} Qm/2. Here the scalar constant c can be dropped as it does not play any role in the optimization. Note that x^{T} Qx is a hyper elliptic paraboloid with the minimum at point m in the N-D space.

We first consider the special case where the QP problem is only subject to equality constraints and we assume m ≤ N, i.e., the number of constraints is smaller than the number of unknowns in x. Then the solution x has to satisfy Ax − b = 0, i.e., it has to be on M hyper planes in the N-D space.

The Lagrangian function of the QP problem is:

L(x, λ) = f(x) + λ^{T} (Ax − b) = \frac{1}{2}x^{T} Qx + c^{T} x + λ^{T} (Ax − b)  (3.79)

To find the optimal solution, we first equate the derivatives of the Lagrangian with respect to both x and λ to zero:

▽_{x}L(x, λ) = Qx + c + A^{T} λ = 0
▽_{λ}L(x, λ) = Ax − b = 0  (3.80)

These two equations can be combined and expressed in matrix form as:

[ \begin{matrix}Q & A^{T} \\ A & 0 \end{matrix} ] [ \begin{matrix}x \\ λ \end{matrix} ] = [ \begin{matrix}−c \\ b \end{matrix} ]  (3.81)

We then solve this system of m + N equations to get the optimal solution x^{∗} and the corresponding λ^{∗}.

Example 3.4

min: f(x_{1}, x_{2}) = x_{1}^{2} + x_{2}^{2} = \frac{1}{2}[x_{1}, x_{2}] [ \begin{matrix}2 & 0 \\ 0 & 2 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = \frac{1}{2}x^{T} Qx
s. t.: Ax = [1, 1] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = x_{1} + x_{2} = b = 1

where

Q = [ \begin{matrix}2 & 0 \\ 0 & 2 \end{matrix} ] , A = [1, 1], c = [ \begin{matrix}0 \\ 0 \end{matrix} ] , b = 1;

[ \begin{matrix}2 & 0 & 1 \\ 0 & 2 & 1 \\ 1 & 1 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ λ \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 1 \end{matrix} ]

Solving this equation we get the solution x_{1}^{∗} = x_{2}^{∗} = 0.5 and λ^{∗} = −1, at which the function f(x_{1}^{∗}, x_{2}^{∗}) = 0.5 is minimized subject to x_{1} + x_{2} = 1.

If M = N, i.e., the number of equality constraints is the same as the number of variables, then the variable x is uniquely determined by the linear system Ax = b, as the intersect of M = N hyper planes, independent of the objective function f(x). Further if M > N, i.e., the system Ax = b is over constrained, and its solution does not exist in general. It is therefore more interesting to consider QP problems subject to both equality and inequality constraints:

min:  f(x) = \frac{1}{2}x^{T} Qx + c^{T} x
s. t.:  Ax ≤ b

3.7 Interior Point Methods

Consider the generic constrained optimization problem in the following form:

min: f(x)
s. t.: \{ \begin{matrix}h(x) = 0 \\ g(x) ≤ 0 or g(x) ≥ 0 \\ x ≥ 0 \end{matrix}

(3.82)

By introducing slack variables s ≥ 0, all inequality constraints can be converted into equality constraints:

g(x) ≤ 0 ⟹ g(x) + s = 0,  g(x) ≥ 0 ⟹ g(x) − s = 0  (3.83)

which can then be combined with the equality constraints and still denoted by h(x) = 0, while the slack variables s are also combined with the original variables and still denoted by x. We assume there are in total M equality constraints and N variables. Now the optimization problem can be reformulated as:

min: f(x)
s. t.: \{ \begin{matrix}h(x) = 0 \\ x ≥ 0 \end{matrix} (3.84)

The Lagrangian is

L(x, λ, μ) = f(x) + λ^{T} h(x) − μ^{T} x  (3.85)

The KKT conditions are

\{ \begin{matrix}▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = 0 & (stationarity) \\ h(x) = 0, x ≥ 0 & (primal feasibility) \\ μ ≥ 0 & (dual feasibility) \\ μ_{j}x_{j} = 0, (j = 1, ⋯, N) or XM1 = 0 & (complementarity) \end{matrix} (3.86)

where

X = [ \begin{matrix}x_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & x_{n} \end{matrix} ]_{N×N} , M = [ \begin{matrix}μ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & μ_{N} \end{matrix} ]_{N×N}  (3.87)

Figure 3.9 Logarithmic Barrier Function

and J_{f} (x) is the Jacobian matrix of h(x):

J_{h}(x) = [ \begin{matrix}\frac{∂h_{1}}{∂x_{1}} & ⋯ & \frac{∂h_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂h_{m}}{∂x_{1}} & ⋯ & \frac{∂h_{m}}{∂x_{n}} \end{matrix} ]_{M×N}  (3.88)

To simplify the problem, the non-negativity constraints can be removed by introducing an indicator function:

I(x) = \{ \begin{matrix}0 & x ≥ 0 \\ ∞ & x < 0 \end{matrix}  (3.89)

which, when used as a cost function, penalizes any violation of the non-negativity constraint x ≥ 0. Now the optimization problem can be reformulated as shown below without the non-negative constants x ≥ 0:

min: f(x) + ∑_{i=1}^{N} I(x_{i})
s. t.: h(x) = 0

(3.90)

However, as the indicator function is not smooth and therefore not differentiable, it is approximated by the logarithmic barrier function which approaches I(x) when the parameter t > 0 approaches infinity:

−\frac{1}{t} ln(x) ⟹^{t→∞} I(x)  (3.91)

Now the optimization problem can be written as:

min: f(x) − \frac{1}{t} ∑_{i=1}^{N} ln x_{i}
s. t.: h(x) = 0

(3.92)

The Lagrangian is:

L(x, λ, μ) = f(x) + λ^{T} h(x) − \frac{1}{t} ∑_{j=1}^{N} ln x_{j}  (3.93)

and its gradient is:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − \frac{1}{t}X^{−1}1  (3.94)

We define

μ = \frac{x}{t} = \frac{1}{t}X^{−1}1, i.e. Xμ = XM1 = \frac{1}{t}  (3.95)

and combine this equation with the KKT conditions of the new optimization problem to get

\{ \begin{matrix}g_{f}(x) + J_{h}^{T}(x)λ − μ = 0 & (stationarity) \\ h(x) = 0 & (primal feasibility) \\ XM1 − 1/t = 0 & (complementarity) \end{matrix}  (3.96)

The third condition is actually from the definition of μ given above, but here it is labeled as complementarity, so that these conditions can be compared with the KKT conditions of the original problem. We see that the two sets of KKT conditions are similar to each other, but with the following differences:

• The non-negativity x ≥ 0 in the primal feasibility of the original KKT conditions is dropped;

• Consequently the inequality μ ≥ 0 in the dual feasibility of the original KKT conditions is also dropped;

• There is an extra term 1/t in complementarity which will vanish when t → ∞.

The optimal solution that minimizes the objective function f(x) can be found by solving the simultaneous equations in the modified KKT conditions (with no inequalities) given above. To do so, we first express the equations in the KKT conditions above as a nonlinear equation system F(x, λ, μ) = 0, and find its Jacobian matrix J_{F} composed of the partial derivatives of the function F(x, λ, μ) with respect to each of the three variables x, λ, and μ:

F(x, λ, μ) = [ \begin{matrix}g_{f}(x) + J_{h}^{T}(x)λ − μ \\ h(x) \\ XM1 − 1/t \end{matrix} ] , J_{F} = [ \begin{matrix}W(x) & J_{h}^{T}(x) & −I \\ J_{h}(x) & 0 & 0 \\ M & 0 & X \end{matrix} ]

(3.97)

where

W(x) = ▽_{x} [g_{f}(x) + J_{h}^{T}(x)λ − μ]
= ▽_{x}g_{f}(x) − ▽_{x} [∑_{i=1}^{M} λ_{i} \frac{∂h_{i}}{∂x_{1}} , ⋯, ∑_{i=1}^{M} λ_{i} \frac{∂h_{i}}{∂x_{N}} ]^{T} = H_{f}(x) − ∑_{i=1}^{M} λ_{i}H_{h_{i}}(x) (3.98)

is symmetric as the Hessian matrices H_{f} and H_{h_{i}}, (i = 1, ⋯, M) are symmetric.

We can now use the Newton-Raphson method to find the solution of the nonlinear equation F(x, λ, μ) = 0 iteratively:

[ \begin{matrix}x_{n+1} \\ λ & n+1 \\ μ & n+1 \end{matrix} ] = [ \begin{matrix}x_{n} \\ λ & n \\ μ & n \end{matrix} ] + α [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ]  (3.99)

where α is a parameter that controls the step size and [δx_{n}, δλ_{n}, δμ_{n}]^{T} is the search direction (Newton direction), which can be found by solving the following equation:

J_{F}(x_{n}, λ_{n}, μ_{n}) [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ] = [ \begin{matrix}W(x_{n}) & J_{h}^{T}(x_{n}) & −I \\ J_{h}(x_{n}) & 0 & 0 \\ M_{n} & 0 & X_{n} \end{matrix} ] [ \begin{matrix}δx_{n} \\ δλ_{n} \\ δμ_{n} \end{matrix} ]

= −F(x_{n}, λ_{n}, μ_{n}) = − [ \begin{matrix}g_{f}(x_{n}) + J_{h}^{T}(x_{n})λ_{n} − μ_{n} \\ h(x_{n}) \\ X_{n}M_{n}1 − 1/t \end{matrix} ]  (3.100)

To get the initial values for the iteration, we first find any point inside the feasible region x_{0} and then μ_{0} = x_{0}/t, based on which we further find λ_{0} by solving the first equation in the KKT conditions in Eq. (3.96).

Actually this equation system above can be separated into two subsystems, which are easier to solve. Pre-multiplying X^{−1} to the third equation:

Mδx + Xδμ = −XM1 + 1/t  (3.101)

we get

X^{−1}Mδx + δμ = −M1 + X^{−1}1/t = −μ + X^{−1}1/t  (3.102)

Adding this to the first equation:

Wδx + J_{h}^{T}(x)δλ − δμ = −g_{f}(x) − J_{h}^{T}(x)λ + μ  (3.103)

we get

(W + X^{−1}M)δx + J_{h}^{T}(x)δλ = −g_{f}(x) − J_{h}^{T}(x)λ + X^{−1}1/t = − ▽_{x} L(x, λ, μ)

(3.104)

Combining this equation with the second one, we get an equation system of two vector variables δx and δλ with symmetric coefficient matrix:

[ \begin{matrix}W + X^{−1}M & J_{h}^{T}(x) \\ J_{h}(x) & 0 \end{matrix} ] [ \begin{matrix}δx \\ δλ \end{matrix} ] = − [ \begin{matrix}▽_{x}L(x, λ, μ) \\ h(x) \end{matrix} ]  (3.105)

Solving this equation system we get δx and δλ, and we can further find δμ by solving the third equation

Mδx + Xδμ = −XM1 − 1/t  (3.106)

to get

δμ = X^{−1}1/t − X^{−1}Mδx − μ  (3.107)

We now consider the two special cases of linear and quadratic programming:

• Linear programming:

min: f(x) = c^{T} x
s. t.: h(x) = Ax − b = 0, x ≥ 0

(3.108)

We have

– g_{f}(x) = ▽_{x}f(x) = c

– J_{h}(x) = ▽_{x}(Ax − b) = A.

– W(x) = ▽_{x}^{2}L(x, λ, μ) = 0. The Lagrangian is

L(x, λ, μ) = c^{T} x + λ^{T} (Ax − b) − μ^{T} x  (3.109)

The KKT conditions are:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = c + A^{T} λ − μ = 0

h(x) = Ax − b = 0
XM1 − 1/t = 0

The search direction of the iteration can be found by solving the this equation system:

[ \begin{matrix}0 & A^{T} & −I \\ A & 0 & 0 \\ M & 0 & X \end{matrix} ] [ \begin{matrix}δx \\ δλ \\ δμ \end{matrix} ] = − [ \begin{matrix}c + A^{T} λ − μ \\ Ax − b \\ XM1 − 1/t \end{matrix} ]  (3.110)

By solving the equation c + A^{T} λ − μ = 0 we get the initial value λ. The Matlab code for the interior point method for the LP problem is listed below:

function x=InterierPointLP(A,c,b,x)
% given A,b,c of the LP problem and inital value of x,
% find optimal solution x that minimizes c'*x
[m n]=size(A);      % m constraints, n variables
z1=zeros(n);
z2=zeros(m);        % zero matrices in coefficient matrix
z3=zeros(m,n);
I=eye(n);           % identity matrix in coefficient matrix
y=ones(n,1);
t=9;                % initinal value for parameter t
alpha=0.3;          % small enough not to exceed boundary
mu=x./t;
lambda=pinv(A')*(mu-c); % initial value of lambda
w=[x; lambda; mu];      % initial values for all three
B=[c+A'*lambda-mu; A*x-b; x.*mu-y/t];
p=[x(1) x(2)];      % initial guess of solution
while norm(B)>10^(-7)

t=t*9;          % increase parameter t
X=diag(x);
M=diag(mu);
C=[z1 A' -I; A z2 z3; M z3' X];
B=[c+A'*lambda-mu; A*x-b; x.*mu-y/t];
dw=-inv(C)*B;   % find search direction
w=w+alpha*dw;   % step forward
x=w(1:n);
lambda=w(n+1:n+m);
mu=w(n+m+1:length(w));
p=[p; x(1), x(2)];
end
scatter(p(:,1),p(:,2));  % plot trajectory of solution
end

Example 3.5 A given linear programming problem shown below is first converted into the standard form:

\begin{matrix}max: & f(x_{1}, x_{2}) = 2x_{1} + 3x_{2} \\ s. t.: & \{ \begin{matrix}2x_{1} + x_{2} ≤ 18 \\ 6x_{1} + 5x_{2} ≤ 60 \\ 2x_{1} + 5x_{2} ≤ 40 \\ x_{1} ≥ 0, x_{2} ≥ 0 \end{matrix} \end{matrix}  ⟹  \begin{matrix}min: & f(x_{1}, x_{2}, x_{3}, x_{4}, x_{5}) = −2x_{1} − 3x_{2} \\ s. t,: & \{ \begin{matrix}2x_{1} + x_{2} + x_{3} = 18 \\ 6x_{1} + 5x_{2} + x_{4} = 60 \\ 2x_{1} + 5x_{2} + x_{5} = 40 \\ x_{i} ≥ 0, (i = 1, ⋯, 5) \end{matrix} \end{matrix}

or in matrix form:

min: f(x) = c^{T} x
s. t. : h(x) = Ax − b = 0, x ≥ 0

with

x = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \\ x_{5} \end{matrix} ] , c = [ \begin{matrix}−2 \\ −3 \\ 0 \\ 0 \\ 0 \end{matrix} ] , A = [ \begin{matrix}2 & 1 & 1 & 0 & 0 \\ 6 & 5 & 0 & 1 & 0 \\ 2 & 5 & 0 & 0 & 1 \end{matrix} ] , b = [ \begin{matrix}18 \\ 60 \\ 40 \end{matrix} ]

where x_{3}, x_{4}, x_{5} are the slack variables.

We choose an initial value x_{0} = [1, 2, 1, 1, 1]^{T} and an initial parameter t = 9, which is scaled up by a factor of 9 in each iteration, We also used full Newton step size of α = 1. As shown in Fig. 3.10, after 8 iterations, the algorithm converged to the optimal solution x^{∗} = [5, 6]^{T} corresponding to

Figure 3.10 Interior Point Method for Linear Programming

the maximum function value f(x^{∗}) = 2x_{1} + 3x_{2} = 28:

(x_{1}  x_{2})  f(x)  error
1 (1.000000e + 00 2.000000e + 00) −8.000000 52.413951
2 (4.654514e + 00 6.346354e + 00) −28.348090 3.080204
3 (5.040828e + 00 5.946973e + 00) −27.922575 0.213509
4 (4.997282e + 00 6.001764e + 00) −27.999856 0.004262
5 (4.999906e + 00 5.999962e + 00) −27.999697 0.000303
6 (4.999989e + 00 5.999996e + 00) −27.999966 0.000034
7 (4.999999e + 00 6.000000e + 00) −27.999996 0.000004
8 (5.000000e + 00 6.000000e + 00) −28.000000 0.000000

At the end of the iteration, we also get the values of the slack variables: x_{3} = 2, x_{4} = x_{5} = 0.

• Quadratic programming:

min: f(x) = \frac{1}{2}x^{T} Qx + c^{T} x
s. t.: h(x) = Ax − b = 0, x ≥ 0

(3.111)

We have

– g_{f}(x) = ▽_{x}f(x) = Qx + c

– J_{h}(x) = ▽_{x}(Ax − b) = A.

– W(x) = ▽_{x}^{2}L(x, λ, μ) = Q.

The Lagrangian is

L(x, λ, μ) = \frac{1}{2}x^{T} Qx + c^{T} x + λ^{T} (Ax − b) − μ^{T} x  (3.112)

The KKT conditions are:

▽_{x}L(x, λ, μ) = g_{f}(x) + J_{h}^{T}(x)λ − μ = Qx + c + A^{T} λ − μ = 0

h(x) = Ax − b = 0
XM1 − 1/t = 0

The equation system for the Newton-Raphson method is:

[ \begin{matrix}Q & A^{T} & −I \\ A & 0 & 0 \\ M & 0 & X \end{matrix} ] [ \begin{matrix}δx \\ δλ \\ δμ \end{matrix} ] = − [ \begin{matrix}Qx + c + A^{T} λ − μ \\ Ax − b \\ XM1 − 1/t \end{matrix} ]  (3.113)

The Matlab code for the interior point method for the QP problem is listed below:

function [x mu lambda]=InteriorPointQP(Q,A,c,b,x)
n=length(c);       % n variables
m=length(b);       % m constraints
z2=zeros(m);
z3=zeros(m,n);
I=eye(n);
y=ones(n,1);
t=9;               % initinal value for parameter t
alpha=0.1;         % small stepsize not to exceed boundary
mu=x./t;
lambda=pinv(A')*(mu-c-Q*x); % initial value of lambda
w=[x; lambda; mu];    % initial value
B=[Q*x+c+A'*lambda-mu; A*x-b; x.*mu-y/t];
while norm(B)>10^(-7)
    t=t*9;         & increase parameter t
    X=diag(x);
    M=diag(mu);
    C=[Q A' -I; A z2 z3; M z3' X];
    B=[Q*x+c+A'*lambda-mu; A*x-b; x.*mu-y/t];
    dw=-inv(C)*B;  % find search direction
    w=w+alpha*dw;  % step forward
    x=w(1:n);
    lambda=w(n+1:n+m);
    mu=w(n+m+1:length(w));
end
end

Example 3.6 A 2-D quadratic programming problem shown below is to minimize four different quadratic functions under the same linear constraints as the linear programming problem in the previous example. In general, the quadratic function is given in the matrix form:

min: f(x) = [x − m]^{T} Q[x − m] = x^{T} Qx + c^{T} x + d
s. t.: h(x) = Ax − b = 0, x ≥ 0

with following four different sets of function parameters:

Q = [ \begin{matrix}4 & −1 \\ −1 & 4 \end{matrix} ] , [ \begin{matrix}2 & 1 \\ 1 & 2 \end{matrix} ] , [ \begin{matrix}2 & 1 \\ 1 & 2 \end{matrix} ] , [ \begin{matrix}3 & −1 \\ −1 & 3 \end{matrix} ]

m = [ \begin{matrix}4 \\ 10 \end{matrix} ] , [ \begin{matrix}9 \\ 5 \end{matrix} ] , [ \begin{matrix}7 \\ 2 \end{matrix} ] , [ \begin{matrix}−1 \\ 6 \end{matrix} ]

As shown in the four panels in Fig. 3.11, the iteration of the interior point algorithm brings the solution from the same initial position at x_{0} = [2, 1]^{T} to the final position for each of the four objective functions f(x), the optimal solution, which is on the boundary of the feasible region in all cases except the third one, where the optimal solution is inside the feasible region, i.e., the optimization problem is not constrained. Also note that the trajectories of the solution during the iteration go straightly from the initial guess to the final optimal solution by following the negative gradient direction of the quadratic function in all cases except in the last one, where it makes a turn while approaching the vertical boundary, due obviously to the significantly greater value of the log barrier function close to the boundary, as shown in Fig. 3.9.

We also note that the dual problem in the SVM algorithm in Eq. (3.52) is in the same form as in Eq. (3.111), i.e., it is a quadratic programming problem that can be solved by the interior point algorithm.

Problems

1. Study carefully the first three sections of the chapter to understand the theoretical background of the constrained optimization problems. Then implement the algorithms discussed in the following sections for both linear and quadratic programming as stated in the following problems.

2. Develop Matlab code for the brute force method for solving an LP problem in the generic form:

maximize  f(x_{1}, ⋯, x_{N}) = c_{1}x_{1} + ⋯ + c_{N}x_{N} = ∑_{i=1}^{N} c_{i}x_{i}

subject to:  \{ \begin{matrix}a_{11}x_{1} + ⋯ + a_{1N}x_{N} = ∑_{i=1}^{n} a_{1i}x_{i} ≤ b_{1} \\ ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ & ⋯ \\ a_{M1}x_{1} + ⋯ + a_{MN}x_{N} = ∑_{i=1}^{N} a_{Mi}x_{i} ≤ b_{M} \\ x_{1} ≥ 0, ⋯, x_{N} ≥ 0 \end{matrix}

Figure 3.11 Interior Point Method for Quadratic Programming

Specifically, find all C_{N+M}^{N} intersections formed by N + M hyperplanes in the N-D space corresponding to the M constraints A_{M×N}x_{N×1} ≤ b_{M×1} and the N conditions x ≥ 0, identify which of them are vertices of the polytope surrounded by these hyperplane, and find the vertex corresponding to the optimal solution that maximizes f(x) = c^{T} x for the given c.

Run your code to solve the LP problem in the example in Section 3.4.

3. Develop Matlab code to implement the simplex algorithm, then apply it to the LP problem in the previous problem, and the maximization problems:

maximize f(x_{1}, x_{2}, x_{3}) = 2x_{1} − 3x_{2} + 4x_{3}
subject to: \{ \begin{matrix}4x_{1} − 3x_{2} + x_{3} ≤ 3 \\ x_{1} + x_{2} + x_{3} ≤ 10 \\ 2x_{1} + x_{2} − x_{3} ≤ 10 \\ x_{1} ≥ 0, x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix}

maximize f(x_{1}, x_{2}, x_{3}) = x_{1} + 2x_{2} − x_{3}
subject to: \{ \begin{matrix}2x_{1} + x_{2} + x_{3} ≤ 14 \\ 4x_{1} + 2x_{2} + 3x_{3} ≤ 28 \\ 2x_{1} + 5x_{2} + 5x_{3} ≤ 30 \\ x_{1} ≥ 0, x_{2} ≥ 0, x_{3} ≥ 0 \end{matrix}

4. Develop code to implement the interior point method, and apply it to both the LP and QP problems in the two examples in Section 3.7.

Part II Regression

In Part II, we will consider various methods for regression analysis, which is a process of supervised learning, with the general goal of establishing a model for the relationship between a set of independent varialbes (stimuli) and another set of dependent variables (responses) of a generic system. Such a model can be symbolically represented as y = f(x, θ), where x = [x_{1}, ⋯, x_{d}]^{T} is a column vector containing d independent variables, y is a scalar for the dependent variable, and column vector θ = [θ_{1}, ⋯, θ_{M}]^{T} contains a set of M parameters of the model. The main task of a supervised learning process is to obtain the unknown model parameters in θ by learning from a set of observed data, called the training set, represented by D = {(x_{n}, y_{n}), (n = 1, ⋯, N)}, containing a set N data samples x_{n} each labeled in some way by the corresponding y_{n}.

Specially, in regression, as one of the supervised learning methods, the model y = f(x, θ) is a specific regression function that maps a data point x, a point in the d-dimensional space, to a scalar output y, represented by a hypersurface in a d + 1 dimensional space spanned by y as well as the d independent variables in x. In particular, the hypersurface is a curve if if d = 1, or a surface if d = 2.

The regression function can be assumed to be either deterministic or probabilistic depending on the specific algorithm. Typically the regression function is hypothesized to take certain form (e.g., linear, higher order polynomial, or exponential) based on some prior knowledge, while the parameters in θ are to be determined by the regression algorithm based on the training set D, which can also be denoted by D = {X, y} in terms of a d × N matrix X = [x_{1}, ⋯, x_{N}] and an N-D vector y = [y_{1}, ⋯, y_{N}]^{T}. Through out the discussion in all the following chapters, we will always assume the data samples in the training set are independently and identically distributed (i.i.d.).

The goal of regression is to obtain the parameters in θ so that the model prediction ŷ_{n} = f(x_{n}, θ) matches optimally the corresponding y_{n} as the ground truth, i.e., the hypersurface of the model best fits all training samples in the d+1 dimensional space.

In this part, we will first in Chapter 4 treat regression as a simple and typical supervised learning process and illustrate some important issues such as underfitting versus overfitting commonly faced in general by all supervised learning methods including not only regression algorithms considered in this Part, but also the classification algorithms to be considered in Parts IV and V. We will consider first various linear and nonlinear regression methods based on deterministic models in Chapters 5 and 6, and then some additional regression methods based on probabilistic models in Chapters 7 and 8, which can also be applied to classification problems.

4 Bias-Variance Tradeoff and Overfitting vs Underfitting

Before discussing various regression algorithms, we first in this chapter consider some guiding philosophy and methodology of supervised learning in general, such as frequentist versus Bayesian approach; the tradeoff and balance between bias and variance errors, or similarly, between overfitting and underfitting of the data; and some general approaches commonly used to address these issues, such as cross-validation and ensemble learning. Although here we consider such issues mostly in the context of regression analysis, we will come back to them over and over again when we discuss other supervised learning algorithms through out the book, all of which need to address the same general issues by the same general approaches, each in its own specific way.

4.1 Supervised Learning as Optimization

In general, supervised learning, including both regression and classification as the two typical types of problems, can be formulated as a system modeling process based on the given training set D = {X, y}, when the independent variable x ∈ X and the dependent variable y ∈ y are treated respectively as the input (stimuli) and output (responses) of a system, of which the behavior in terms of the relationship between such input and output is to be modeled by a generic function y = f(x, θ) parameterized by θ).

In general, a supervised learning problem can be formulated as the framework illustrated in Fig. 4.1, with the goal of constructing a mathematical model, symbolized by function f(x, θ) parameterized by θ, in such a way that its output ŷ matches the labeling y corresponding to the input x in the training set D in some sense. Typically the form of the model f(x, θ) is assumed to be known, which is either a specific function in regression, or certain specific classification algorithm (e.g., statistic model or a neural network) in classification). In any case, the model parameters in θ are to be determined optimally in some sense, such as the discrepancy between the actual output ŷ, the prediction of the model, and the desired output y, the ground truth given in the training set is minimized. In other words, a learning problem is typically solved as an optimization problem.

Specifically, in regression analysis, the model y = f(x, θ) is simply a function with independent variables x and dependent variable y, while in pattern

Figure 4.1 A General Model for Supervised Learning

classification (or object recognition) (to be discussed in Part IV), the model y = f(x, θ) represents a specific classification method, by which a pattern described by the independent variable x as its features is assigned to one of the classes {C_{1}, ⋯, C_{K}} indicated by the corresponding dependent variable y. As both regression and classification can be considered as a general supervised learning process, much of the discussion in this chapter for various regression algorithms is also highly relevant to the classification algorithms to be considered in the future chapters.

The goal of a regression algorithm is for the regression model to optimally fit the observed data, so that ideally it reflects all variations due to the valid signal in the data, but at the same time remains unaffected by the inevitable observation noise also existing in the data. In general, all learning algorithms aiming at modeling a set of observed data face the common dilemma between over and under trusting the observed data contaminated by noise, and they all need to address the same issue of how to best distinguish between signal and noise in the data, so that they neither overfit the data, i.e., overly affected by noise, nor underfit data, i.e., not matching the valid signal variation sufficiently closely. This issue of making a proper tradeoff between overfitting and underfitting will need to be addressed by various means in different learning algorithms, as we will see in this and future chapters.

The problem of regression analysis can be addressed from different philosophical viewpoints. First, from the frequentist point of view, the unknown model parameters in θ are fixed deterministic variables and their values are to be estimated based on the training set. A typical method based on this viewpoint is the least squares (LS) method for both linear and nonlinear regression methods. Alternatively, from the point of view of Bayesian inference, the model parameters in θ are random variables. We can estimate their prior probability distribution p(θ) based on some prior knowledge without observing any data. If no such prior knowledge is available, we can simply assumed p(θ) is a uniform distribution, i.e., all possible values of θ are equally likely. Once the training data X are available and observed, we can estimate the posterior probability of the model parameters based on Bayes’ theorem:

p(θ|X) = \frac{p(X|θ)p(θ)}{p(X)} ∝ p(X|θ)p(θ)  (4.1)

In general, the posterior p(θ|X) is a narrower distribution than the prior p(θ), and therefore a more accurate description of the model parameters, due to the reduced uncertainty of the parameters based on additional information gained from the observed data. The output ŷ = f(x, θ) of such a regression model based Bayesian inference is no longer a deterministic value, but a random variable described by its probability distribution in terms of its mean and variance. Typical methods based on this point of view include Bayesian linear regression, logistic regression and Gaussian process regression.

Based on either of these two viewpoints, different regression methods can be used to find the parameters θ for the model function ŷ = f(x, θ), such as those listed below.

• Least squares (LS) method:

This frequentist method measures how well the regression function models the training data by the residual defined as r(θ) = y − ŷ(θ), defined as the difference between the model prediction ŷ(θ) = f(x, θ) and the ground truth labeling value y corresponding to x, for each of the N data points in the observed data:

r(θ) = [ r_{1} ; ⋮ ; r_{N} ] = [ y_{1} − ŷ_{1} ; ⋮ ; y_{N} − ŷ_{N} ] = [ y_{1} − f(x_{1}, θ) ; ⋮ ; y_{N} − f(x_{N}, θ) ] = y − ŷ (4.2)

Ideally, for the residual to be zero, r(θ) = 0, we find the M model parameters as components of θ by solving this nonlinear equation system of N equations of M unknowns in θ. However, as in general N ≫ M, this system is overconstrained without a solution. We therefore can only use least squares method to find the optimal solution that minimizes the sum of squared error (SSE):

ε(θ) = \frac{1}{2}||r||^{2} = \frac{1}{2}r^{T} r = \frac{1}{2} ∑_{n=1}^{N} r_{n}^{2} = \frac{1}{2} ∑_{n=1}^{N}(y_{n} − f(x_{n}, θ))^{2}  (4.3)

Here the coefficient 1/2 is included for mathematical convenience. When divided by the total number of samples N, this SSE becomes the average of all N squared errors, and the error becomes the mean squared error (MSE). Minimizing either SSE or MSE produces the same results.

• Maximum likelihood estimate (MLE):

This Bayesian inference method measures how well the regression function models the training data in terms of the likelihood L(θ|D) of the model parameter θ based on the observed dataset D = {X, y}, which is proportional to the conditional probability of D given θ:

L(θ|D) ∝ p(D|θ) = p(y|X, θ)  (4.4)

The optimal model parameters in θ can be found as those that maximize L(θ|D) or equivalently p(y|X, θ).

We assume the residual r = y − f(x, θ) is a zero-mean random variable (without loss of generality) with a normal probability density function (pdf):

p(r) = p(y − f(x, θ)) = N (r, 0, σ^{2}) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y − f(x, θ))^{2}}{2σ^{2}} )
= N (y, f(x, θ), σ^{2}) = p(y|x, θ)  (4.5)

The justification for this assumed normal pdf is that it has the greatest entropy (Section B.2.1), or maximum uncertainty among all possible pdfs with the same variance, i.e., given σ^{2}, the assumed normal pdf imposes least amount of constraint and thereby minimum bias.

The zero-mean pdf of r = y − f(x, θ) can also be considered as the pdf of y with mean f(x, θ), the conditional pdf p(y|x, θ) of y given θ as well as x, based on the assumption that y and x are related by y = f(x, θ). Then we can find the likelihood of θ given the data samples in the training set, all assumed to be independently and identically distributed:

L(θ|D) = p(y|X, θ) = ∏_{n=1}^{N} p(y = y_{n}|x_{n}, θ)

= ∏_{n=1}^{N} \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y_{n} − f(x_{n}, θ))^{2}}{2σ^{2}} )  (4.6)

or, equivalently and for mathematical convenience, minimize the negative log likelihood function:

−l(θ|D) = − log L(θ|X, y) = − ∑_{n=1}^{N} log p(y = y_{n}|x_{n}, θ)  (4.7)

= − ∑_{n=1}^{N} log [\frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(y_{n} − f(x, θ))^{2}}{2σ^{2}} )]

= \frac{N}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∑_{n=1}^{N} ||r||^{2}  (4.8)

The first term and the coefficient 1/σ^{2} in the second term are both independent of θ and can therefore be dropped. We note that this result is equivalent to that in Eq. (4.3), i.e., we can find the optimal θ by either maximizing the likelihood L(θ|D), or equivalently minimizing the SSE ε(θ).

We further note that negative log likelihood −l(θ|D) = − log L(θ|X, y) in the equation above can also be interpreted differently as the cross entropy (Section B.2.2) between an unknown ground truth probability P and the model probability Q:

H(P, Q) = − ∑_{i} P_{i} log Q_{i} = −E_{P} [log Q]  (4.9)

We rewrite Eq. (4.7) as the average of the logarithm of the model distribution Q = p(y|x, θ) over all N observed samples in the dataset D = {X, y}:

−l(θ|D) = − ∑_{n=1}^{N} log p(y = y_{n}|x_{n}, θ) = −E_{(X,y)} [log p(y = y_{n}|x_{n}, θ)]

(4.10)

and realize that this is the cross entropy H(P, Q) estimated based on the N samples, which can be minimized for model Q to match the ground truth P as closely as possible.

• Maximum A Posteriori (MAP)

This is also a Bayesian inference method that measures how well the regression function models the training data by the posterior probability of the parameter θ given in Eq. (4.1), proportional to the product of the likelihood L(θ|D) ∝ p(D|θ) and the prior p(θ). If no prior knowledge about θ is available, then p(θ) is a uniform distribution, then MAP is equivalent to MLE. However, if certain prior knowledge regarding θ does exist, and the prior is not uniform, then the posterior better characterizes θ than the likelihood and MAP may produce better result than MLE.

We conclude that regression analysis, and in general many supervised learning algorithms to be considered in the following chapters, can be treated as an overconstrained optimization problem with many more observed data points in the training set than the number of unknown parameters in the model. Following different philosophical view points, this optimization problem can be addressed differently, such as minimizing the sum of squared errors, maximizing the likelihood of the model parameters, or minimizing the negative likelihood, i.e., the cross entropy between the model and unknown ground truth distributions of the given dataset, to be discussed in details in the following sections.

More generally, some of the concepts and methods in regression analysis discussed in the following sections are also shared by many supervised learning algorithms for classification. For example, the general method of neural network is conceptually based on some ideas very different from regression, but both of these two methods can be described mathematically as a function y = f(x, θ) that models the relationship between the independent variable x and dependent variable y. While this function is called the regress function in this chapter, it is also a way to describe the complex behavior of a multilayer learning network to be discussed in Section 18.2

4.2 Bias-Variance Tradeoff

Before discussing specifically various regression algorithms, we first address an important issue that commonly appears in not only regression problems as discussed below, but also in general all supervised learning problems based on training dataset (or simply training set) such as classification.

In a generic regression problem, the goal is to model the unknown relationship between a dependent variable y = f(x) and d independent variables in x = [x_{1}, ⋯, x_{d}]^{T}, by a hypothesis regression function f̂(x), based on a set of training data D = {(x_{n}, y_{n}) n = 1, ⋯, N} containing N observed data samples. As the observation is inevitably contaminated by some random noise e, both the observed value modeled by y = f(x) + e and the regression model f̂(x) need to be treated as random variables. We need to consider all possible errors of different types in the regression analysis as the consequence of such noisy data.

We first assume without loss of generality that the random noise e has a zero mean E[e] = 0 and variance σ^{2}, and it is independent of both f and f̂:

Var[e] = E[(e − E[e])^{2}] = E[e^{2}] = σ^{2}
E[e f] = E[e] E[f], E[e f̂] = E[e] E[f̂]  (4.11)

To measure how well the model f̂(x) fits the noisy data y = f(x) + e, we define the mean squared error (MSE) between them (with x dropped for simplicity):

E[(y − f̂)^{2}] = E[(f + e − f̂)^{2}] = E [[(f − Ef̂) + (Ef̂− f̂) + e]^{2}]
= E [(f − Ef̂)^{2} + (Ef̂− f̂)^{2} + e^{2} + 2e(f − Ef̂) + 2e(Ef̂− f̂) + 2(f − Ef̂)(Ef̂− f̂)]
= E[(f − Ef̂)^{2}] + E[(Ef̂− f̂)^{2}] + E(e^{2})
+2 E[e(f − Ef̂)] + 2 E[e(Ef̂− f̂)] + 2 E[(Ef̂− f̂)(f − Ef̂)]
= E[(f − Ef̂)^{2}] + E[(Ef̂− f̂)^{2}] + E[e^{2}]  (4.12)

The last equality is due to the fact that all three cross terms are zero:

\{ \begin{matrix}E[e(f − Ef̂)] = E[e] E[f − Ef̂] = 0 \\ E[e(Ef̂− f̂)] = E[e] E[Ef̂− f̂] = 0 \\ E[(Ef̂− f̂)(f − Ef̂)] = (Ef̂− Ef̂)(f − Ef̂) = 0 \end{matrix}  (4.13)

Now the total mean squared error in Eq. (4.12) can be expressed in terms of three different types of errors:

Mean Squared Error = Variance Error + Bias Error + Irreducible Error (4.14)

where

• Bias Error: E[(f − Ef̂)^{2}] = (f − Ef̂)^{2} measures how well the estimated function f̂ models the ground truth function f;

• Variance Error: E[f̂− Ef̂]^{2} = Var[f̂] measures the variation of the model;

• Irreducible Error E[e^{2}] = σ^{2} measures the inevitable observation noise.

The variance error and bias error are illustrated in Fig. 4.2. Specially comparing the top right and the bottom left cases we see that under the same total MSE and irreducible error σ^{2}, the bias error and the variance error are complementary to each other, i.e., if one is higher, the other is lower, depending on the specific model function f̂. This relationship is called the bias-variance tradeoff, which is related to the common issue of overfitting versus underfitting in all

Figure 4.2 Bias-Variance Tradeoff

supervised learning problems in general as we will see in many of the learning algorithms through out the book, such as support vector machine in Chapter 14, back-propagation network in Chapter 18, and reinforcement learning in Chapter 20.

• Overfitting:

To match the training data closely, a complex model may be needed to take into consideration all detailed variations in the data which may be caused by the noise in the data. Such a model tends to have a low variance error, but may suffer from high bias error, as it may be overly affected by the random noise such as some outliers, as shown in the top-right case in Fig. 4.2. Such a model does not generalize well across multiple datasets contaminated differently by noise.

• Underfitting:

On the other hand, to avoid being overly affected by the noise, a simple model may be used with the possible consequence that it may not match the detailed variation due to the valid signal in the data sufficiently closely. Such a model tends to have low bias error as it is insensitive to the varying noise across datasets, but it may suffer from high variance error as it may not match all valid signal variations. This is the lower-left case in Fig. 4.2.

Overfit Underfit

Variance error High

Low

Bias error  Low  High

(4.15)

These two possibilities are illustrated by the examples in Fig. 4.3. As both overfitting and underfitting result in poor prediction capability, they

Figure 4.3 Under vs. Overfitting

Figure 4.4 Under and Overfitting in Regression and Classification

need to be avoided by making a proper tradeoff between bias-error/underfitting and variance-error/overfitting. This is one of the most important issues in general in all supervised learning algorithms aiming at modeling a system and predicting its behaviors, including regression for predicting the output value ŷ = f(x) as a function of a given data point x, and classification for providing a categorical class labeling of a given point x.

The issue of overfitting versus underfitting is further illustrated in Fig. 4.4. In the case of regression (left panel), the data points are modeled by two different regression functions, and in classification, the 2-D space is partitioned into two regions corresponding to two classes by two different decision boundaries. In both cases, the given datasets are overfitted by the blue curve underfitted by the red curve.

• Underfitting:

The simple linear regression function or decision boundary (red) may underfit the data as some legitimate variations in the data may be missed. This is a problem if all data points reliably reflect some complex variations due to valid signal in the data.

• Overfitting:

The complex regression model (e.g., a high order polynomial) or decision boundary (blue) may overfit the data if it is heavily affected by the variation due to observation noise instead of valid signal in the data. This is a problem if the data samples are taken from a simple process, but are fitted by an unnecessarily complex model due to the noise, i.e., a violation of the principle of Occam’s razor.

Therefore we face the challenge of how to make a proper tradeoff between over and underfittings, so that the algorithm can produce an optimal model that fits the variations in the given data, in the sense that it maximally fits the valid signal in the data, but at the same time minimally affected by the noise also in the data. Various methods to address this challenge will be considered in the following section and through out the discussion of many supervised learning algorithms in the following chapters.

4.3 Cross-Validation

In the case when the entire training set is used for training to take full advantage of the available data which may be limited to start with, it is impossible to distinguish between variations due to valid signal and those due to random observation noise, and to judge whether the model produced by the learning algorithm in question either over or underfits the given data. This problem can be addressed by the method of cross-validation. If the dataset available for training the algorithm is of sufficient size, it can be partitioned into two or more subsets each used to either train or test the algorithm. This method is based on the simple idea that the valid signal variation should be shared by all sub-datasets, while they are contaminated differently by the noise due to its random nature. If the algorithm is trained by one subset of data but tested by another, then its performance measured by certain error (calculated differently depending on the specific learning problem and algorithm used, e.g., residual for regression, or rate of error for classification) can be used to assess the algorithm in terms of whether it suffers from the overfitting problem.

Cross-validation has different variations depending on how the available dataset is partitioned.

• Exhaustive methods: the algorithm is trained and tested based on the available dataset divided in all possible ways into multiple subsets. Typically, the leave-p-out cross-validation uses p out of the total n sample points in the dataset to test the algorithm after it is trained by the remaining n − p samples. To achieve unbiased result, we would need to repeat this process C_{p}^{n} times to exhaust all possible ways to choose p samples out of n. To avoid repeating too many times when n is large, we may need to consider the special case of either leave-pair-out (p = 2) or leave-one-out (p = 1).

• Non-exhaustive methods: Typically the K-fold cross-validation divides the dataset into K (e.g., K = 4 or 5) equal-sized subsets, of which K − 1 subsets are used for training while the remaining one is reserved for testing. This process is repeated K times each based on a different testing subset and then the average of the K errors is used to check if the algorithm suffers overfitting. Specially we can simply let K = 2, or we can also randomly divide the available data into two subsets each for either training or testing, and calculate the average of the errors obtained while the process is repeated multiple times.

The method of cross-validation method can be used to detect and assess the algorithm in question to see whether or how much it over or under fits the given data, it has certain limitations. First, the method requires much increased computational complexity. Second, the method is effective only when the size of the training set available is sufficiently large and can therefore be divided into multiple subsets of meaningful sizes. If the size of the available dataset if small, as may be the case in some applications, the results generated by the cross-validation may not be reliable and meaningful.

4.4 Regularization and Ensemble Learning

The method of cross-validation can detect and assess overfitting problems, but it does not correct the problem by itself. Some additional approaches can be considered to address this problem.

First, we consider regularization, which is an method widely used in many supervised learning algorithms (including both regression and classification) to deal with the issue of overfitting versus underfitting. Typically this is done by adjusting certain hyperparameters of the model to make a proper tradeoff between the two extremes of over and underfittings. In particular, if the problem at hand is ill-conditioned (see Section A.8.5), then regularization can be highly effective so that the algorithm is less sensitive to noise in the dataset and therefore less overfitting, i.e., it is more generalizable while still able to capture the essential nature and behavior of the valid signal. We will show how regularization is actually implemented, such as the ridge regression in Section 5.2, and many other learning algorithm to be considered in future chapters.

Second, we consider the general method of Ensemble learning, a general approach that take advantage of multiple learning algorithms, each considered as a hypothesis, trained on multiple datasets, so that the resulting model can best fits the give data. This method is based on the idea that a single learning algorithm trained on a single dataset, here called a learner, may be weak as it may suffer from the problem of under vs overfitting, but a set of multiple such weak learners (either identical or different) can be trained on multiple datasets and combined in some way to become a strong learner. In general such a ensemble learning method comes with two flavors depending on how the weak learners are combined to form a strong learner.

• Bootstrap Aggregation (Bagging)

This is a two-step process including bootstrapping in which a set of independent weak learners are trained in parallel by different subsets as random subsamples from the total dataset (with replacement), and aggregation in which a strong learner combines the results by the individual weak learners with equal weights, such as their average for regression, or their majority vote for classification. The weak learners can be either identical (homogeneous) or different (heterogeneous). In the latter case, the method is also called stacking, as the collection of all such weak learners based on different algorithms and learning in parallel are treated as a stack. As we will see in Section 16, the popular method of random forest is a bagging algorithm that combines multiple decision tree classifiers for better classification performance.

• Boosting

This is a multi-step sequential process in which the weak learners are trained sequentially, in the sense that each learner is trained on different subset of data, based on the learner in the previous step, thereby continuously improving the overall performance of the strong learner. As we will see in Section 13.4, the method of adaptive boosting or AdaBoost is such a boosting algorithm for classification.

5 Linear Regression

5.1 Linear Least Squares (LLS) Regression

In linear regression, the relationship between the dependent variable y and the d independent variables {x_{1}, ⋯, x_{d}} is modeled by a linear hypothesis function:

ŷ = f(x, θ) = f(x, w, b) = b + ∑_{i=1}^{d} w_{i}x_{i} = b + x^{T} w  (5.1)

parameterized by θ containing b and w = [w_{1}, ⋯, w_{d}]^{T} for the coefficients or weights in the linear function. The goal of a linear regression algorithm is to estimated the model parameters in θ basied on the training set D = {X, y} containing a set of N given data points in X = [x_{1}, ⋯, x_{N}] and the corresponding labelings in y = [y_{1}, ⋯, y_{N}]^{T}.

For notational convenience, we redefine both x and w so that they become augmented d + 1 dimensional vectors: x = [x_{0}, x_{1}, ⋯, x_{d}]^{T} with an extra component x_{0} = 1 and w = [w_{0}, w_{1}, ⋯, w_{d}]^{T} with an extra component w_{0} = b. Now the linear model above can be expressed more concisely as

ŷ = f(x, w) = x^{T} w  (5.2)

Geometrically, the linear regression function f(x) = w^{T} x is a straight line if d = 1, a plane if d = 2, or a hyperplane if d > 2, in the d + 1 dimensional space spanned by y as well as {x_{1}, ⋯, x_{d}}. The function has an intercept f(0) = w_{0} = b along the y dimension (when x_{1} = ⋯ = x_{d} = 0), and a normal vector [w_{1}, ⋯, w_{d}, −1]^{T}. When the regression function is set to zero, i.e., the regression line or plane is thresholded by the plane y = 0, it becomes an equation f(x) = x^{T} w = 0, representing a point if d = 1, a straight line if d = 2, a plane or hyperplane if d ≥ 3, in the d-dimensional space spanned by {x_{1}, ⋯, x_{d}}, as shown in Fig. 5.1 for d = 2.

We denote by X the matrix containing the N augmented data points x_{n} = [x_{0n} = 1, x_{1n}, ⋯, x_{dn}]^{T} (n = 1, ⋯, N) as its column vectors:

X = [ \begin{matrix}1 & ⋯ & 1 \\ x_{11} & ⋯ & x_{1N} \\ ⋮ & ... & ⋮ \\ x_{d1} & ⋯ & x_{dN} \end{matrix} ]_{(d+1)×N} = [x_{1}, ⋯, x_{N}]  (5.3)

Figure 5.1 Linear Regression as a Binary Classifier

and its transpose can be written as:

X^{T} = [ \begin{matrix}1 & x_{11} & ⋯ & x_{d1} \\ ⋮ & ⋮ & ... & ⋮ \\ 1 & x_{1N} & ⋯ & x_{dN} \end{matrix} ]_{N×(d+1)} = [1, \underline{x}_{1}, ⋯, \underline{x}_{d}]  (5.4)

where 1 = [1, ⋯, 1]^{T}, and \underline{x}_{i} = [x_{i1}, ⋯, x_{iN}]^{T} (i = 1, ⋯, d) is an Ndimensional vectors containing the ith components of all N data vectors in the dataset. Now the linear regression problem is simply to find the model parameter w, so that the model prediction

ŷ = X^{T} w  (5.5)

matches the ground truth labeling y, so that the residual of the linear model r = y −X^{T} w is zero, i.e., we need to solve the following linear system containing N equations of d + 1 unknowns:

r = y − ŷ = y − X^{T} w = 0  (5.6)

If the number of data samples is equal to the number of unknown parameters, i.e., N = d + 1, then X is square and invertible matrix (assuming all N samples are independent and therefore X has a full rank), and the equation above can be solved to get the desired weight vector:

w = (X^{T})^{−1}y  (5.7)

However, if there are many more data samples than the unknown parameters, i.e., N ≫ d + 1, then X is not square and invertible, then X^{T} w = y is an overdetermined linear system without a solution and the prediction ŷ = X^{T} w will not match the ground truth labeling y in general. In this case we need to find an optimal solution w^{∗} that minimizes the SSE (proportional to MSE):

ε(w) = \frac{1}{2} ∑_{n=1}^{N} r_{n}^{2} = \frac{1}{2}||r||^{2} = \frac{1}{2}||y − X^{T} w||^{2}
= \frac{1}{2}y^{T} y − y^{T} X^{T} w + \frac{1}{2}w^{T} XX^{T} w  (5.8)

To find the optimal w that minimizes ε(w) as the objective function, we set its gradient to zero and get the normal equation:

g_{ε}(w) = \frac{d}{dw} ε(w) = XX^{T} w − Xy = 0, i.e. XX^{T} w = Xy  (5.9)

Solving this equation for w, we get the optimal weight vector:

w^{∗} = (XX^{T})^{−1}Xy = (X^{T})^{−}y  (5.10)

where (X^{T})^{−} = (XX^{T})^{−1}X is the (d + 1) × N pseudo-inverse (Section A.6.1) of the N × (d + 1) matrix X^{T}. This method for estimating the regression model parameters is called ordinary least squares (OLS).

Having found the parameter w for the linear regression model ŷ = x^{T} w, we can predict the outputs corresponding to any unlabeled test dataset X_{∗} = [x_{1∗}, ⋯, x_{M∗}]:

ŷ_{∗} = f_{∗} = X_{∗}^{T} w^{∗} = [1, \underline{x}_{1}, ⋯, \underline{x}_{N}] [ w_{0}^{∗} ; w_{1}^{∗} ; ⋮ ; w_{d}^{∗} ] = w_{0}^{∗}1 + ∑_{i=1}^{d} w_{i}^{∗}\underline{x}_{i}  (5.11)

as a linear combination of 1 and \{\underline{x}_{1}, ⋯, \underline{x}_{d}\}.

We can show geometrically that the estimated ŷ = x^{T} w^{∗} based on the optimal w^{∗} is indeed the best fit of the training data. We first rewrite the normal equation in Eq. (5.9) as:

XX^{T} w^{∗} − Xy = X (X^{T} w^{∗} − y) = X (ŷ − y)

= Xr = [ \begin{matrix}1 & ⋯ & 1 \\ x_{11} & ⋯ & x_{1N} \\ ⋮ & ... & ⋮ \\ x_{d1} & ⋯ & x_{dN} \end{matrix} ] r = [ 1^{T} ; \underline{x}_{1}^{T} ; ⋮ ; \underline{x}_{d}^{T} ] r = 0  (5.12)

The last equality indicate that the residual vector r is perpendicular to each of the d + 1 vectors of X^{T} = [1, \underline{x}_{1}, ⋯, \underline{x}_{d}]:

1^{T} r = ∑_{n=1}^{N} r_{n} = 0, \underline{x}_{1}^{T} r = ⋯ = \underline{x}_{N}^{T}r = 0  (5.13)

Figure 5.2 Explained, Residual and Total Sum of Squares

i.e., r is perpendicular to the space spanned by \{1, \underline{x}_{1}, ⋯, \underline{x}_{d}\}, in which all possible estimated ŷ = x^{T} w based on any arbitrary w reside. We can therefore conclude that ||r|| = ||ŷ − x^{T} w^{∗}|| based on w^{∗} is the smallest error among all possible ||r′|| = ||ŷ − x^{T} w|| based on other w ≠ w^{∗}, i.e., ŷ = x^{T} w^{∗} is indeed the best with minimum residual ||r||^{2}. This is illustrated in Fig. 5.2 for d = 1.

we first derive some properties of the model based on its output ŷ = X^{T} w = X^{T} (XX^{T})^{−1}Xy:

||ŷ||^{2} = ŷ^{T} ŷ = (X^{T} (XX^{T})^{−1}Xy)^{T} (X^{T} (XX^{T})^{−1}Xy)
= (y^{T} X^{T} (XX^{T})^{−1}X) (X^{T} (XX^{T})^{−1}Xy)
= y^{T} X^{T} (XX^{T})^{−1}Xy = y^{T} ŷ  (5.14)

based on which we also get the following results:

r^{T} ŷ = (y − ŷ)^{T} ŷ = y^{T} ŷ − ŷ^{T} ŷ = 0  (5.15)
||r||^{2} = r^{T} r = (y − ŷ)^{T} (y − ŷ) = y^{T} y − 2y^{T} ŷ + ŷ^{T} ŷ
= y^{T} y − y^{T} ŷ = y^{T} (y − ŷ) = y^{T} r  (5.16)
||ŷ||^{2} + ||r||^{2} = y^{T} ŷ + y^{T} r = y^{T} (ŷ + r) = y^{T} y = ||y||^{2}  (5.17)

We next consider how to quantitatively measure the goodness of the linear regression model y = w^{T} x, i.e., how well it fits the given dataset. To do so, we first define a vector ȳ = ȳ 1 of which all components are the mean of {y_{1}, ⋯, y_{N}}:

ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n} = \frac{1}{N} y^{T} 1  (5.18)

and then define the following three sums of squares of the components (squared norm) of the three vectors ŷ − ȳ, y − ŷ, and y − ȳ:

• Explained sum of squares (ESS) that measures the variation in the data (with respect to its mean) explained by the regression model:

ESS = ||ŷ − ȳ||^{2} = ∑_{n=1}^{N}(ŷ_{n} − ȳ)^{2}  (5.19)

• Residual sum of squares (RSS) that represents the variation in the data not explained by the data (due to noise or discrepancy between the data and the model):

RSS = ||r||^{2} = ||y − ŷ||^{2} = ∑_{n=1}^{N}(y_{n} − ŷ_{n})^{2}  (5.20)

• Total sum of squares (TSS) for the total data variation:

T SS = ||y − ȳ||^{2} = ∑_{n=1}^{N}(y_{n} − ȳ)^{2}  (5.21)

We can show that the total sum of squares is the sum of the explained sum of squares and the residual sum of squares:

T SS = ||y − ȳ||^{2} = ||y − ȳ + ŷ − ŷ||^{2} = ||(ŷ − ȳ) + (y − ŷ)||^{2}
= ||ŷ − ȳ||^{2} + 2(y − ŷ)^{T} (ŷ − ȳ) + ||y − ŷ||^{2}
= ||ŷ − ȳ||^{2} + 2r^{T} (ŷ − ȳ) + ||r||^{2}
= ESS + 2r^{T} ŷ − 2r^{T} ȳ + RSS = ESS + RSS  (5.22)

The last equality is due to the fact that the two middle terms are both zero (Eqs. (5.13) and (5.15)):

\{ \begin{matrix}r^{T} ȳ = ∑_{n=1}^{N} r_{n}ȳ = ȳ∑_{n=1}^{N} r_{n} = 0 \\ r^{T} ŷ = (y − ŷ)^{T} ŷ = y^{T} ŷ − ŷ^{T} ŷ = 0 \end{matrix}

We also see that all three vectors above are perpendicular to vector 1:

(ŷ − ȳ)^{T} 1 = (ŷ − y + y − ȳ)^{T} 1 = (ŷ − y)^{T} 1 + (y − ȳ)^{T} 1 = 0
(y − ŷ)^{T} 1 = ∑_{n=1}^{N}(y_{n} − ŷ_{n}) = ∑_{n=1}^{N} r_{n} = 0

(y − ȳ)^{T} 1 = ∑_{n=1}^{N}(y_{n} − ȳ) = N(ȳ − ȳ) = 0  (5.23)

(The second equation above is due to Eq. (5.13).) The geometry of these three vectors and the simple relationship in Eq. (5.22) is illustrated in Fig. 5.2 for d = 1.

We can now measure the goodness of the regression model based on the three sums of squares above. Given the total sum of squares T SS, if the model residual RSS is small, then ESS is large, indicating most of the variation in the data can be explained by the regression model, i.e., it fits the data well and is therefore a goo model. Otherwise if RSS is large, the model is not good. Specifically, the goodness of the model can be measured by the coefficient of determination, denoted by R^{2} (R-squared), as the percentage of variation in the data that is explained by the model:

R^{2} = \frac{ESS}{T SS} = \frac{T SS − RSS}{T SS} = 1 − \frac{RSS}{T SS}  (5.24)

In general, 0 < R^{2} < 1. In the ideal case where the data is perfectly modeled by the linear regression function f(x) with zero error r = y −ŷ = 0, then RSS = 0, T SS = ESS, and R^{2} = 1, i.e., the data is completely explained by the model. In a nonideal case with r = y − ŷ ≠ 0, R^{2} < 1.

How well a linear regression function models a given dataset depends on its nature such as how closely the variables in the dataset are related to each other and how noisy the dataset is, which can be measured in terms of the correlation between the variables. In general, the more correlated the variables, the better the dataset can be modeled, as shown below for the special case of d = 1. In this case how closely the two variables x and y in the 1-D dataset {(x_{n}, y_{n}) (n = 1, ⋯, N)} are correlated can be measured by correlation coefficient defined as (Section B.1.3):

ρ = \frac{σ_{xy}^{2}}{\sqrt{σ_{x}^{2} σ_{y}^{2}}} = \frac{σ_{xy}^{2}}{σ_{x} σ_{y}}  (5.25)

where σ_{x}^{2} and σ_{y}^{2} are the estimated variances of x and y:

σ_{x}^{2} = \frac{1}{N} ∑_{n=1}^{N}(x_{n} −x̄)^{2} = \frac{1}{N} ∑_{n=1}^{N} x_{n}^{2} −x̄^{2}, σ_{y}^{2} = \frac{1}{N} ∑_{n=1}^{N}(y_{n} −ȳ)^{2} = \frac{1}{N} ∑_{n=1}^{N} y_{n}^{2} −ȳ^{2}

(5.26)

with

x̄ = \frac{1}{N} ∑_{n=1}^{N} x_{n}, ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n}  (5.27)

and σ_{xy}^{2} is their estimated covariance:

σ_{xy}^{2} = \frac{1}{N} ∑_{n=1}^{N}(x_{n} − x̄)(y_{n} − ȳ) = \frac{1}{N} ∑_{n=1}^{N} x_{n}y_{n} − x̄ȳ  (5.28)

The correlation coefficient is a value between −1 and 1, it can be considered as the normalized covariance of the two variables x and y, indicating how much they are either positively or negatively correlated:

• ρ = 1: maximum positive correlation

• 0 < ρ < 1: positive correlation

• ρ = 0: not correlated at all • −1 < ρ < 0: negative correlation

• ρ = −1: maximum negative correlation

The model parameters of this 1-D linear regression function y = w_{0} + w_{1} x can be obtained based on Eq. (5.10), and expressed in terms of the variances and covariance of the two variables:

w = [ \begin{matrix}w_{0} \\ w_{1} \end{matrix} ] = (XX^{T})^{−1}Xy

= ([ \begin{matrix}1 & ⋯ & 1 \\ x_{1} & ⋯ & x_{N} \end{matrix} ] [ \begin{matrix}1 & x_{1} \\ ⋮ & ⋮ \\ 1 & x_{N} \end{matrix} ])^{−1} [ \begin{matrix}1 & ⋯ & 1 \\ x_{1} & ⋯ & x_{N} \end{matrix} ] [ y_{1} ; ⋮ ; y_{N} ]

= [ \begin{matrix}N & ∑_{n=1}^{N} x_{n} \\ ∑_{n=1}^{N} x_{n} & ∑_{n=1}^{N} x_{n}^{2} \end{matrix} ]^{−1} [ \begin{matrix}∑_{n=1}^{N} y_{n} \\ ∑_{n=1}^{N} x_{n}y_{n} \end{matrix} ] = [ \begin{matrix}1 & x̄ \\ x̄ & σ_{x}^{2} + x̄^{2} \end{matrix} ]^{−1} [ ȳ ; σ_{xy}^{2} + x̄ȳ ]
= \frac{1}{σ_{x}^{2}} [ \begin{matrix}σ_{x}^{2} + x̄^{2} & −x̄ \\ −x̄ & 1 \end{matrix} ] [ ȳ ; σ_{xy}^{2} + x̄ȳ ] = [ \begin{matrix}ȳ − σ_{xy}^{2}/σ_{x}^{2} x̄ \\ σ_{xy}^{2}/σ_{x}^{2} \end{matrix} ]  (5.29)

Based on these model parameters we get the linear regression model:

ŷ = w_{0} + w_{1}x = ȳ − \frac{σ_{xy}^{2}}{σ_{x}^{2}} x̄ + \frac{σ_{xy}^{2}}{σ_{x}^{2}} x = ȳ + \frac{σ_{xy}^{2}}{σ_{x}^{2}} (x − x̄)  (5.30)

The coefficient of determination R^{2} for the goodness of the regression model is closely related to the correlation coefficient ρ for correlation of the given data (independent of the model). Consider

\frac{1}{N} RSS = \frac{1}{N} ||y − ŷ||^{2} = \frac{1}{N} ∑_{n=1}^{N}(y_{n} − w_{0} − w_{1}x_{n})^{2}

= \frac{1}{N} ∑_{n=1}^{N}(y_{n} − (ȳ − x̄w_{1}) − w_{1}x_{n})^{2} = \frac{1}{N} ∑_{n=1}^{N}[(y_{n} − ȳ) − w_{1}(x_{n} − x̄)]^{2}

= \frac{1}{N} ∑_{n=1}^{N}[(y_{n} − ȳ)^{2} − 2w_{1}(y_{n} − ȳ)(x_{n} − x̄) + w_{1}^{2}(x_{n} − x̄)^{2}]

= σ_{y}^{2} − 2w_{1}σ_{xy}^{2} + w_{1}^{2}σ_{x}^{2} = σ_{y}^{2} − 2\frac{σ_{xy}^{2}}{σ_{x}^{2}} σ_{xy}^{2} + \frac{σ_{xy}^{4}}{σ_{x}^{4}} σ_{x}^{2} = σ_{y}^{2} − \frac{σ_{xy}^{4}}{σ_{x}^{2}}
= σ_{y}^{2} (1 − \frac{σ_{xy}^{4}}{σ_{x}^{2}σ_{y}^{2}} ) = σ_{y}^{2}(1 − ρ^{2})  (5.31)

We see that if the two variables x and y are more correlated with ρ closer to 1, then the residual error RSS = ||r||^{2} is smaller and R^{2} is larger, indicating the data can be better fitted by the linear regression model.

Example 5.1 In 1929, Edwin Hubble published a paper “A relationship between distance and radial velocity among extra-galactic nebulae” (Proceedings of the

Figure 5.3 Linear Regression between recessional velocity and distance of nebulae, based on Hubble’s origional data (right) and modern data (left)

National Academy of Sciences of the United states of America (PANS), Vol. 15), where the linear regression method is used to model the relationship between the distance from Earth in megaparsec (Mpc = 3.26 × 10^{6} light years) and the recessional velocity (in kilometer/second) of nebulae (galaxies), as shown in the left panel of Fig. 5.3. Based on this plot, Hubble estimated the slope of the linear regression, now called the Hubble constant

H_{0} = \frac{recessional velocity}{distance} = \frac{v}{D}

to be 500 km/sec/Mpc. The estimated value of the Hubble constant has been continuously revised using different methods based on data with much wider range in distance collected by progressive technology. The most recent estimations of the Hubble constant are roughly in the range around 70 km/sec/Mpc, de-

pending on the specific methods used (see https://en.wikipedia.org/wiki/Hubble%27s_law).

One of such recent results is shown in the right panel of Fig. 5.3 (“Hubble’s Diagram and Cosmic Expansion”, R. P. Kirshner, PANS, Vol. 101 (1), Dec. 2003, Copyright (2003) National Academy of Sciences, U.S.A.).

The simple linear relationship between the recessional velocity v and distance D has led to some very profound results such as the expansion of the universe, the Big Bang as the beginning of the expansion, and the age of the universe since the Big Bang, which can be estimated as (assuming the expansion has been linear with H_{0} = 72 since the Big Bang, and c = 3 × 10^{5} km/sec):

t_{0} = \frac{1}{H_{0}} = \frac{D}{v} = \frac{1 Mpc}{72 km/sec} = \frac{3.26 × 10^{6} × 3 × 10^{5}}{72} = 13.6 × 10^{9}years

Example 5.2 Find a linear regression function y = w_{0}+w_{1} x to fit the following N = 6 points:

\begin{matrix}x & −3.4 & −2.1 & −0.8 & 0.3 & 1.7 & 2.5 \\ y & −0.76 & −1.04 & 1.75 & 1.82 & 3.17 & 3.15 \end{matrix}

Based on the data, we get

ȳ = 1.348, σ_{x}^{2} = 4.22, σ_{y}^{2} = 2.85, σ_{xy}^{2} = 3.27, ρ = \frac{σ_{xy}^{2}}{σ_{x}σ_{y}} = 0.94

The augmented data array is:

X = [ \begin{matrix}1 & 1 & 1 & 1 & 1 & 1 \\ −3.4 & −2.1 & −0.8 & 0.3 & 1.7 & 2.5 \end{matrix} ]

Solving the over-constrained equation system

y = [ \begin{matrix}−0.76 \\ −1.04 \\ 1.75 \\ 1.82 \\ 3.17 \\ 3.15 \end{matrix} ] = X^{T} w = [ \begin{matrix}1 & −3.4 \\ 1 & −2.1 \\ 1 & −0.8 \\ 1 & 0.3 \\ 1 & 1.7 \\ 1 & 2.5 \end{matrix} ] [ \begin{matrix}w_{0} \\ w_{1} \end{matrix} ]

we get

w = X^{−}y = (XX^{T})^{−1}X y

= [ \begin{matrix}0.13 & 0.15 & 0.16 & 0.17 & 0.19 & 0.20 \\ −0.12 & −0.07 & −0.02 & 0.02 & 0.08 & 0.11 \end{matrix} ] [ \begin{matrix}−0.76 \\ −1.04 \\ 1.75 \\ 1.82 \\ 3.17 \\ 3.15 \end{matrix} ] = [ \begin{matrix}1.58 \\ 0.78 \end{matrix} ]

and the linear regression model function y = w_{0} +w_{1} x = 1.58+0.78 x, a straight line with intercept w_{0} = 1.58, slope w_{1} = 0.78, and normal direction [0.78, −1]^{T}, as shown in Fig. 5.4. The LS error is ||r|| = ||y − ŷ|| = 1.38. We further get

ȳ = [1.348, 1.348, 1.348, 1.348, 1.348, 1.348]^{T}
ŷ = X^{T} w = [−1.054, −0.047, 0.961, 1.813, 2.898, 3.518]^{T}

and

ESS = 15.21, RSS = 1.91, T SS = 17.12, R^{2} = 0.89

We can also check that

\frac{1}{N} RSS = σ_{y}^{2}(1 − ρ^{2}) = 0.318

Example 5.3 The two datasets given in the table below are fitted by two linear regression functions as shown in the left and right panels in Fig. 5.5.

\begin{matrix}x & −3.0 & −2.5 & −2.0 & −1.5 & −1.0 & −0.5 & 0.0 \\ y_{1} & −2.67 & −3.76 & −0.64 & −0.93 & −0.01 & −1.42 & 3.38 \\ y_{2} & 2.73 & −9.38 & −1.44 & −9.86 & −2.82 & −7.29 & 8.31 \end{matrix} · · · · · ·

Figure 5.4 Linear Regression Example

Figure 5.5 Linear Regressions with Different R^{2} and ρ

· · · · · · \begin{matrix}0.5 & 1.0 & 1.5 & 2.0 & 2.5 & 3.0 \\ −0.47 & 1.77 & 3.67 & 6.40 & 4.49 & 8.82 \\ 0.98 & 10.18 & 8.58 & 10.86 & 0.77 & 0.20 \end{matrix}

w_{0} w_{1} ESS RSS T SS R^{2}  ρ
dataset1 1.43 1.72 134.71 30.18 164.89 0.817 0.904
dataset2 0.91 1.92 166.98 443.92 610.89 0.273 0.523

Example 5.4 Fig. 5.6 shows a set of N = 30 data points {(x_{i}, y_{i}), i = 1, ⋯, 30} fitted by a 1-D linear regression function y = w_{0} + w_{1}x, a straight line, with w_{0} = 1.81, w_{1} = 3.22. The correlation coefficient is ρ = 0.98, and ESS = 734.78, RSS = 32.98, T SS = 767.77, R^{2} = 0.96. Fig. 5.7 further shows three more examples with different correlation coefficient values ρ = 0.982, 0.110, 0.989, indicating the two variables are highly positively correlated, not correlated, and highly negatively correlated, respectively.

Example 5.5 Fig. 5.6 shows a set of N = 90 data points {(x_{i}, y_{i}), i = 1, ⋯, 90} fitted by a 2-D linear regression function y = w_{0} + w_{1}x_{1} + w_{2}x_{2},

Figure 5.6 Example Showing y, ŷ and ȳ

Figure 5.7 Examples of Different Correlation Coefficients

with w_{0} = 2.82, w_{1} = 1.85, w_{2} = 4.31. This is a plane as shown in Fig. 5.8 from two perspective view angles. Here ESS = 677.00, RSS = 124.48, T SS = 802.47 and R^{2} = 0.84.

Finally we note that the linear regression method can be used for binary classification, if the regression function y = f(x), as a hyperplane in the d + 1 dimensional space spanned by {x_{1}, ⋯, x_{d}, y}, is thresholded by a constant C, e.g., C = 0. The resulting equation f(x) = C defines a hyperplane in the d dimensional space spanned by {x_{1}, ⋯, x_{d}}, which partitions the space into two parts in such a way that all points on one side of the hyperplane satisfy f(x) > C, while all points on the other side satisfy f(x) < C. We see that the regression function is a binary classifier that separates all points x in the d dimensional space into two classes C_{+} and C_{−}, depending on whether f(x) is greater or smaller than C. Now each y_{n} corresponding to x_{n} in the given dataset D = {X, y} can be treated as a label indicating x_{n} belong to class C_{+} if y_{n} > C, or C_{−} if y_{n} < C, and the regression problem becomes a binary classification problem. This binary classifier is illustrated in Fig. 5.1 for C = 0.

Figure 5.8 Linear Regression of a 2-D Dataset

5.2 Ridge Regression

The linear regression method considered in previous section may be prone to noise and suffer from overfitting, if the problem is ill-conditioned, when the data samples in the dataset are barely independent and therefore ill-behaved. See more detailed discussion in Section A.8.5. In such a case, matrix XX^{T} still has a full rank but it is close to singularity, i.e., some of its eigenvalues may be very close to zero, and the corresponding eigenvalues of its inverse (XX^{T})^{−1} may take very large values, causing some components of the model parameter w = (XX^{T})^{−1}Xy to also take very large values. Now any minor change in the dataset (in either x or y) due to noise may be amplified by w to cause a major change in the model output ŷ = w^{T} x. In other words, the regression model becomes highly prone to noise, i.e., it may overfit a given dataset with a large variance error thereby causing poor generalizability.

This issue of ill-conditioning can be illustrated in the 2-D case, where a linear regression function f(x) = w^{T} x, a 3-D plane, is to be determined to fit a set of given data points {(x_{n}, y_{n}), n = 1, ⋯, N}. If two variables x_{1} and x_{2} in the dataset are highly correlated, i.e., they are narrowly distributed in the 2-D space around a straight line (similar to the case in either the first or third panel in Fig. 5.7), then the value of |ρ| is large and close to 1, and some eigenvalues of XX^{T} take small values close to 0, and the 2-D regression problem becomes illconditioned as it degenerates into a 1-D problem. If y is linearly related to x_{1} and x_{2}, then all data points {(x_{n}, y_{n}), n = 1, ⋯, N} are closely distributed around a straight line in the 3-D space which can be fitted by any 3-D plane through the line with drastically different orientations (rotated differently around the 3-D line as an axis). Consequently it becomes difficult to determine a unique 3-D plane to fit the data as it may change drastically due to minor variations in y_{n} or x_{n}.

Figure 5.9 2-D Linear Regression, Well-conditioned (left) versus Ill-Conditioned (right)

Example 5.6 Fig. 5.9 compares two different datasets one well behaved while the other ill-behaved. In the first case shown in the left panel, the two independent variables x_{1} and x_{2} are widely distributed in the 2-D plane, i.e., they are uncorrelated with a small ρ = −0.322, and the matrix XX^{T} has two large eigenvalues λ_{1} = 12.34 and λ_{2} = 25.44. This is a well-conditioned problem that results in a reliable regression plane as shown. In the second case shown in the right panel x_{1} and x_{2} are narrowly distributed around a straight line in the 2-D plane, i.e., they are highly correlated with a large ρ = 0.99, and one of the two eigenvalues λ_{1} = 0.09 and λ_{2} = 43.24 of matrix XX^{T} is close to zero. This is a ill-conditioned problem that results in an unstable regression plane that can be drastically affected (rotating around a straight line in the 3-D space) due to noise in the data.

To address the ill-conditioned problem in linear regression, the method of ridge regression can be used to regularize the problem by imposing some additional constraints to prevent w from taking large values while minimizing the squared error SSE in Eq. (5.8). To do so, we construct an objective function J(w) to be minimized that contains not only the SSE term ε = ||r||^{2}/2, but also a regularization or penalty term λ||w||^{2}/2 to discourage large values in w:

J(w) = \frac{1}{2}||y − X^{T} w||^{2} + \frac{λ}{2}||w||^{2}  (5.32)

so that the optimal w that minimizes this objective function has a small norm ||w|| as well as a small SSE. This method is called weight decay and the hyperparameter λ is called the weight decay parameter. Similar to Eq. (5.9), we set its gradient to zero:

\frac{d}{dw} J(w) = XX^{T} w − Xy + λw = (XX^{T} + λI)w − Xy = 0  (5.33)

and solve the resulting equation to get

w^{∗} = (XX^{T} + λI)^{−1}Xy  (5.34)

By adjusting the hyperparameter λ, we can make a proper tradeoff between stability of the result and its accuracy in terms of the goodness of the model measured by R^{2}):

• If λ is small, the result is more accurate but less stable as it is more prone to noise, i.e., it tends to overfit the data with a large bias error but small variance.

• If λ is large, the result is less affected by noise and therefore more stable, but it is less accurate, i.e., it tends to underfit the data with a small bias error but a large variance error.

In general, to prevent overfitting, regularization by means of including in the objective function a constraint term to impose some additional constraints over the model parameters is commonly used in many supervised learning algorithms for both regression and classification, including the classification methods of support vector machine in Section 14 and the back propagation neural network in Section 18.2.

Example 5.7 The method of ridge regression is applied to the same ill-conditioned dataset in the right panel in Fig. 5.9, where the two independent variables x_{1} and x_{2} are highly correlated with ρ = 0.99. As the result, matrix XX^{T} is close to singularity with one of its two eigenvalues λ_{1} = 43.24 and λ_{2} = 0.09 close to zero, i.e., the matrix is close to singularity and the problem is ill-conditioned.

When the regular linear regression (λ = 0) is applied, it produces the results shown in the top two panels in Fig. 5.10 for the dataset before and after the dependent variable y is slightly perturbed. We see that the orientations of the two resulting regression planes differ drastically, due to the perturbance in y. But when the ridge regression method (with λ = 1) is applied, it produces more stable results as shown in the bottom two panels of the figure. We see that the same perturbation in the dataset caused only a minor change in the orientation of the regression plane.

The cost of such more stable results is their accuracy, as the model goodness values R^{2} for the dataset before and after the perturbance are reduced from 0.894 and 0.915 for the regular regression to 0.857 and 0.876 respectively for the ridge regression.

This example shows that the stability of the solution of an ill-conditioned problem can be improved with the cost of lower accuracy (smaller R^{2}), and tradeoff between stability (low bias error) and accuracy (low variance error) can be made by adjusting parameter λ.

Figure 5.10 Unstable Results Generated by Regression applied to Ill-Conditioned Dataset (top) and Stable Results Generated by Ridge Regression (bottom)

5.3 Regression Based on Basis Functions

The method of linear regression considered previously can be generalized to model nonlinear relationships between the dependent variable y and the independent variables in x = [x_{1}, ⋯, x_{d}]^{T} by a regression function as a linear combination of K nonlinear basis functions of x:

ŷ = f(x) = ∑_{k=1}^{K} w_{k} φ_{k}(x, θ) = φ(x)^{T} w  (5.35)

where φ_{k}(x, θ), (k = 1, ⋯, K) is one of the basis function parameterized by θ that span the function space in which y resides, and φ(x) = [φ_{1}(x), ⋯, φ_{K}(x)]^{T} is a K-D vector. Some of the commonly used basis functions are listed below and plotted in Fig. 5.11 for K = 7.

• Power function (top-left):

φ_{k}(x) = x^{k−1}  (5.36)

This is the (k−1)th power of x, and the regression function f(x) is (K−1)th order polynomial of x.

• Sinusoidal function (top-right):

φ_{k}(x, ω) = sin((k − 1)ωx), or φ_{k}(x, ω) = cos((k − 1)ωx) (5.37)

Figure 5.11 Basis Functions for Regression

This is the (k − 1)th harnomic of a sinusoidal function of x, and the regression function f(x) is the first K terms of the sine or cosine series expansion of the given function.

• Gaussian function (bottom-left):

φ_{k}(x, c_{k}, a) = exp(a||x − c_{k}||^{2})  (5.38)

This is a member of a family of functions called radial basis functions (RBFs), each centered at c_{k}.

• Sigmoid function (bottom-right):

φ_{k}(x, c_{k}, a) = \frac{1}{1 + e^{−a||x−c_{k}||}}  (5.39)

In practice, choosing a proper type of basis functions may be a trial-and-error process depending on the specific dataset.

The original d-dimensional data space spanned by x_{1}, ⋯, x_{d} is now converted to a K-dimensional space spanned by the K basis functions φ_{1}(x), ⋯, φ_{K}(x). Applying such a regression model to each data point x_{n}, we get

ŷ_{n} = ∑_{k=1}^{K} w_{k} φ_{k}(x_{n}) = φ(x_{n})^{T} w (n = 1, ⋯, N)  (5.40)

These N equations can be written in matrix form:

ŷ = [ y_{1} ; ⋮ ; y_{N} ] = [ φ^{T} (x_{1}) ; ⋮ ; φ_{K}^{T}(x_{N}) ] w = Φ^{T} w  (5.41)

where Φ is a K × N matrix:

Φ = [φ(x_{1}), ⋯, φ(x_{N})] = [ \begin{matrix}φ_{1}(x_{1}) & ⋯ & φ_{1}(x_{N}) \\ ⋮ & ... & ⋮ \\ φ_{K}(x_{1}) & ⋯ & φ_{K}(x_{N}) \end{matrix} ]  (5.42)

Typically there are many more sample data points than the basis functions, i.e., N ≫ K.

If, specially, φ(u) = u is an identity function, then Eq. (5.35) becomes the same as Eq. (5.2) for linear regression. Also, as the nonlinear model ŷ = Φ^{T} w in Eq. (5.41) is in the same form as the linear model ŷ = X^{T} w in Eq. (5.2), it can be solved the same way as in Eq. (5.10), by simply replacing X by Φ:

w^{∗} = (ΦΦ^{T})^{−1}Φy = (Φ^{T})^{−}y  (5.43)

where (Φ^{T})^{−} = (ΦΦ^{T})^{−1}Φ is the K × N pseudo-inverse of the N × K matrix Φ^{T}. Now we get the output of the regression model

ŷ = Φ^{T} w = Φ^{T} Φ^{−}y = Φ^{T} (ΦΦ^{T})^{−1}Φy  (5.44)

The Matlab code segment below shows the essential part of the algorithm, where x(n) is the nth data sample x_{n} out of the total of N, y(x) is the vector y = [y_{1}, ⋯, y_{N}]^{T} with y_{n} = f(x_{n}), and c(k) contains the parameters for the kth basis function ϕ_{k}(x) = ϕ(x, c(k)) represented by phi in the code.

     for n=1:N
         for k=1:K
             Phi(k,n)=phi(x(n),c(k));
         end
     end
     w=pinv(Phi')*y(x);      % weight vector by LS method
     yhat=Phi'*w;            % reconstructed function

Example 5.8 The plots in Figs. 5.12 through 5.15 show the approximations of a given function based on a set of N samples using linear regression based on the four different types of basis functions. We see that as the number of basis functions K increases, the approximation becomes more accurate.

Example 5.9 The linear regression method based on basis functions is shown in Fig. 5.16, where a 2-D function f(x) = f(x_{1}, x_{2}) (left) is to be approximated based on the function values in {y_{1}, ⋯, y_{N}} at a set N sample points {x_{1}, ⋯, x_{N}} taken from a grid in the 2-D space (right). The regression is based on K 2-D Gaussian basis functions centered at different locations in the 2-D space shown in Fig. 5.17 for K = 9.

The regression results based on different numbers of basis functions are shown in Fig. 5.18. Again, we see that the approximation becomes more accurate as the value of K increases.

In addition to the different types of basis functions considered above, we fur-

148  Linear Regression

Figure 5.12 Regression based on Power Basis Figure 5.13 Regression based on Sinusoidal Basis

ther consider the following basis functions:

φ_{k}(x, w_{k}, b_{k}) = g (w_{k}^{T} x + b_{k}) = g (∑_{i=1}^{d} w_{ki}x_{i} + b_{k}) = g (∑_{i=0}^{d} w_{ki}x_{i}) (5.45)

where g(x), called activation function, can take some different forms such as a

5.3 Regression Based on Basis Functions 149

Figure 5.14 Regression based on Gaussian Basis Figure 5.15 Regression based on Sigmoid Basis

sigmoid or rectified linear function (rectified linear unit or ReLU),

g(x) = \{ \begin{matrix}1/(1 + e^{−x}) & sigmoid function \\ max(0, x) = \{ \begin{matrix}0 & x < 0 \\ x & x ≥ 0 \end{matrix} & ReLU \end{matrix}  (5.46)

Now the regression function in Eq. (5.35) can be written as below for multiple

Figure 5.16 A Function and Its Samples in 2-D

Figure 5.17 Gaussian Basis in 2-D

outputs :

ŷ_{j} = ∑_{k=1}^{K} w_{jk}^{o} φ_{k}(x, θ_{k}) = ∑_{k=1}^{K} w_{jk}^{o} g (∑_{i=0}^{d} w_{ki}^{h}x_{i}) (j = 1, ⋯, m) (5.47)

which can be interpreted as the output of a three-layer neural network shown in Fig. 5.19 containing an input layer of d nodes, a middle or hidden layer of K nodes with weights w_{ki}^{h}, (i = 0, ⋯, d, k = 1, ⋯, K), and an output layer of

Figure 5.18 Regression Results based on Different Number of Basis Functions

Figure 5.19 A Network Model for Regression

m nodes with weights w_{jk}^{o}, (k = 1, ⋯, K, j = 1, ⋯, m). These nodes are also called neurons, as they mimick the biological neurons in the brain.

As shown in Fig. 5.19, the nodes of the input layer take the d components of a data point x from the dataset one at a time, while each of the K nodes of the hidden layer calculates their linear combination w_{k}^{T} x, and, together with a bias term b_{k}, generates the basis function φ_{k}(x) = g (w_{k}^{T} x + b_{k}) as its output. Finally, each node of the output layer takes these K basis function from the K hidden layer and generates as its output the regression function ŷ = f(x) = ∑_{k=1}^{K} w_{k}φ_{k}(x) as the model of the dataset.

This network depends on two sets of weights, those represented by w_{ki}^{h} for the hidden nodes, and those represented by w_{jk}^{o} (found in Eq. (5.43)) for the output nodes. When there are multiple nodes in the output layer, they can each approximate a different function. This three-layer network is of particular interest in the general method of artificial neural networks to be considered in Chapter 17.

Example 5.10 The method of regression based on basis functions is used to approximate the same 2-D function f(x) = f(x_{1}, x_{2}) in the previous example, as a linear combination of K basis functions φ_{k}(x) = w_{k}^{T} x + b_{k} with each parameterized by w_{k} and b_{k} (randomly generated).

The Matlab code segment below shows the essential part of the program for this example, where f is a arbitrary function to be approximated.

a=@(x) x.*(x>0);      % ReLU activation function, or
a=@(x) 1/(1+exp(-x)); % Sigmoid activation function
g=@(x,w,b)a(w'*x+b);  % the basis function
[X,Y]=meshgrid(xmin:1:xmax, ymin:1:ymax); % define 2-D points
nx=size(X,2);         % number of samples in first dimension
ny=size(Y,1);         % number of samples in second dimension
N=nx*ny;              % total number of data samples
Phi=zeros(N,K);      % K basis functions evaluated at N sample points
W=1-2*rand(2,K);      % random initialization of weights
b=1-2*rand(1,K);      % and biases for K basis functions
n=0;
for i=1:nx
    x(1)=X(1,i);      % first component
    for j=1:ny
        x(2)=Y(j,1);  % second component
        n=n+1;        % of the nth sample
        Y(i,j)=func(x); % function evalued at the sample
        for k=1:K       % basis functions evaluated at the sample
            Phi(k,n)=g(x,W(:,k),b(k));
        end
    end
end
y=reshape(Y,N,1);     % convert function values to vector
w=pinv(Phi')*y;       % find weights by LS method
yhat=Phi'*w;          % reconstructed function
Yhat=reshape(yhat,m,n); % convert vector y to 2-D to compare w/ Y

Figs. 5.20 through 5.23 show the basis functions based on both the sigmoid and ReLU activation functions, and the corresponding regression results. We

Figure 5.20 Sigmoid Basis for 2-D Regression

note that as more basis functions are used, the more accurately the function is approximated.

5.4 Bayesian Regression

The regression algorithms previously considered frequentist methods that treat all data and parameters in a regression model as deterministic variables. However, regression can also be viewed in light of Bayesian inference, where all data and parameters in the regression model are treated as random variables. In this section, we consider two such methods, the maximum likelihood estimation (MLE) and the maximum a posterior (MAP) estimation. While the MLE method turns out to be equivalent to the least squares method, the MAP method will generate results including not only the estimated model parameter w but also the variance of the estimate as a confidence measurement. These methods can also be used for nonlinear regression based on basis functions.

Here we assume the model parameter w in the linear regression function ŷ = f(x, w) = x^{T} w is a random vector. Consequently, the regression function f(x, w), as a function of w, is also random, so is the residual r = y−ŷ = y−X^{T} w

Figure 5.21 Regression based on Sigmoid Basis

based on the training set D = {X, y}. Now we have

y = ŷ + r = f(X) + r = X^{T} w + r  (5.48)

We assume w has a zero-mean normal prior pdf:

p(w) = N (w, 0, Σ_{w}) ∝ exp (−\frac{1}{2}w^{T} Σ_{w}^{−1}w) ,  (5.49)

based on our desire that the weights in w should take values around zero, instead of large values on either positive or negative, to avoid potential overfitting. Note that here the normalizing factor independent of w is dropped for simplicity. We further assume that all components of the residual r are independent of each other and they have the same variance, i.e., Σ_{r} = σ_{r}^{2}I, and r has a zero-mean normal distribution:

p(r) = N (r, 0, σ_{r}^{2}I) ∝ exp (−\frac{1}{2σ_{r}^{2}} r^{T} r) = exp (−\frac{1}{2σ_{r}^{2}} ||r||^{2})  (5.50)

As a linear transformation of w, the regression function ŷ = X^{T} w also has a zero-mean Gaussian distribution, and so is y = X^{T} w + r as the sum of two Gaussians.

Figure 5.22 ReLU Basis for 2-D Regression

Given the observed data D = {X, y}, we get its likelihood:

L(w|D) ∝ p(D|w) = p(y|X, w) = p(X^{T} w + r) = N (y, X^{T} w, σ_{r}^{2}I)
∝ exp (−\frac{1}{2σ_{r}^{2}} (y − X^{T} w)^{T} (y − X^{T} w)) = exp (−\frac{1}{2σ_{r}^{2}} ||r||^{2})(5.51)

We see that the likelihood is a normal pdf of y (the same normal pdf but shifted by X^{T} w), which is the same as that of r given above. Now we can used the maximum likelihood estimation (MLE) method to find the optimal parameter w^{∗} by which the likelihood L(w|D) is maximized, or equivalently the following negative log likelihood is minimized:

−l(w|X, y) = − log L(w|X, y) = \frac{||r||^{2}}{2σ_{r}^{2}} ∝ ||r||^{2}  (5.52)

This is the same as the squared error ε(w) defined in Eq. (5.8) to be minimized by the LS method previously considered. We therefore see that the maximum likelihood method and the least squares method are equivalent to each other.

We can further find the posterior p(w|D) of the model parameter w given the observed data D, as the product of prior p(w) and likelihood L(w|D) ∝

Figure 5.23 Regression based on ReLU Basis

p(y|X, w):

p(w|D) = p(w|X, y) = \frac{p(D|w) p(w)}{p(D)} = \frac{p(y|X, w) p(w)}{p(y|X)}
∝ p(y|X, w) p(w) = N (y, X^{T} w, σ_{r}^{2}I) N (w, 0, Σ_{w})
∝ exp [−\frac{1}{2} (\frac{1}{σ_{r}^{2}} (y − X^{T} w)^{T} (y − X^{T} w) + w^{T} Σ_{w}^{−1}w)]
∝ exp [−\frac{1}{2} (\frac{1}{σ_{r}^{2}} y^{T} y − \frac{2}{σ_{r}^{2}} (Xy)^{T} w + \frac{1}{σ_{r}^{2}} w^{T} XX^{T} w + w^{T} Σ_{w}^{−1}w)]
∝ exp [−\frac{1}{2} (w^{T} (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1}) w − \frac{2}{σ_{r}^{2}} (Xy)^{T} w)]  (5.53)

As before, here all constant scaling factors (irrelevant to w) are dropped for simplicity. As both the prior and the likelihood are Gaussian, this this posterior, when normalized, is also Gaussian in terms of the mean m_{w/d} and covariance Σ_{w/d}:

p(w|D) = N (w, m_{w/d}, Σ_{w/d}) ∝ exp [−\frac{1}{2} ((w − m_{w/d})^{T} Σ_{w/d}^{−1}(w − m_{w/d}))]
∝ exp [−\frac{1}{2} (w^{T} Σ_{w/d}^{−1}w − 2m_{w/d}^{T}Σ_{w/d}^{−1}w)](5.54)

with the following covariance and mean:

Σ_{w/d} = (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1})^{−1}
m_{w/d} = \frac{1}{σ_{r}^{2}} Σ_{w/d}Xy = (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy  (5.55)

Now we can find the optimal model parameter w^{∗} = m_{w/d} equal to the mean of the Gaussian posterior at which it is maximized. This solution is called the maximum a posteriori (MAP) estimate of the model parameter w.

We can now predict the function value of any unlabeled test data point x_{∗} based on the conditional probability distribution p(f_{∗}|x_{∗}, D) of the regression model ŷ_{∗} = f_{∗} = f(x_{∗}) = x_{∗}^{T} w, which, as a linear transformation of the random parameters in w, is also Gaussian (Section B.1.5).

p(f_{∗}|x_{∗}, D) = p(x_{∗}^{T} w|x_{∗}, X, y) = N (f_{∗}, x_{∗}^{T} m_{w/d}, x_{∗}^{T} Σ_{w/d}x_{∗})  (5.56)

with the following mean and variance:

E[f_{∗}|x_{∗}, X, y] = x_{∗}^{T} m_{w/d} = x_{∗}^{T} (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy
Var[f_{∗}|x_{∗}, X, y] = x_{∗}^{T} Σ_{w/d}x_{∗} = x_{∗}^{T} (\frac{1}{σ_{r}^{2}} XX^{T} + Σ_{w}^{−1})^{−1} x_{∗} (5.57)

Furthermore, given a set of M unlabeled test data points in X_{∗} = [x_{1∗}, ⋯, x_{M∗}]^{T}, we can find

ŷ_{∗} = f_{∗} = f(x_{∗}) = [ f(x_{1∗}) ; ⋮ ; f(x_{M∗}) ] = [ x_{1∗}^{T} ; ⋮ ; x_{M∗}^{T} ] w = [x_{1∗}, ⋯, x_{M∗}]^{T} w = X_{∗}^{T} w

(5.58)

As a linear combination of Gaussian w, f_{∗} = X_{∗}^{T} w is also Gaussian:

p(f_{∗}|X_{∗}, D) = p(f_{∗}|X_{∗}, X, y) = N (f_{∗}, X_{∗}^{T} m_{w/d}, X_{∗}^{T} Σ_{w/d}X_{∗})  (5.59)

with the following mean and covariance:

E[f_{∗}|X_{∗}, X, y] = E[X_{∗}^{T} w] = X_{∗}^{T} E[w] = X_{∗}^{T} m_{w/d} = X_{∗}^{T} (XX^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Xy (5.60)
Cov[f_{∗}|X_{∗}, X, y] = E[f_{∗}f_{∗}^{T}] = E[X_{∗}^{T} ww^{T} X_{∗}] = X_{∗}^{T} E[ww^{T}]X_{∗} = X_{∗}^{T} Σ_{w/d}X_{∗} (5.61)

The mean E[f_{∗}|X_{∗}, X, y] above can be considered as the estimated function value ŷ_{∗} = f_{∗} = X_{∗}^{T} w at the test points in X_{∗}, while the variance Var[f_{∗}|X_{∗}, X, y] as the confidence or certainty of the estimation.

Such a result produced by the Bayesian method in terms of the mean and covariance of f_{∗} = ŷ_{∗} as a random vector is quite different from the result produced by a frequentist method such as the linear least square regression in terms of an estimated value of ŷ_{∗} as a deterministic variable. However, in the special case of σ_{r} = 0, i.e., the residual r is no longer treated as a random variable, then Eq. (5.60) above becomes the same as Eq. (5.11) produced by linear least square regression, in other words, the linear method method can be considered as a special case of the Bayesian regression when the residual is deterministic.

The method considered above is based on the linear regression model f(x) = x^{T} w, but it can also be generalized to nonlinear functions, same as in the case of regression based on basis functions discussed in Section 5.3. To do so, the regression function is assumed to be a linear combination of a set of K nonlinear mapping functions φ_{k}(x) (k = 1, ⋯, K):

y = f(x) + e = ∑_{k=1}^{K} w_{k} φ_{k}(x) + r = w^{T} ϕ(x) + r = ϕ^{T} (x)w + r  (5.62)

where

w = [w_{1}, ⋯, w_{K}]^{T} , ϕ(x) = [φ_{1}(x), ⋯, φ_{K}(x)]^{T}  (5.63)

For example, some commonly used mapping functions are listed in Eq. (5.45).

Applying this regression model to each of the N observed data points in D = {(x_{n}, y_{n}), (n = 1, ⋯, N)}, we get ŷ_{n} = ∑_{k=1}^{K} w_{k} φ_{k}(x_{n}) with residual r_{n} = y_{n} − ŷ_{n}, which can be written in matrix form:

y = [ y_{1} ; ⋮ ; y_{N} ]_{N×1} = [ \begin{matrix}φ_{1}(x_{1}) & ⋯ & φ_{K}(x_{1}) \\ ⋮ & ... & ⋮ \\ φ_{1}(x_{N}) & ⋯ & φ_{K}(x_{N}) \end{matrix} ]_{N×K} [ w_{1} ; ⋮ ; w_{K} ]_{K×1} + [ r_{1} ; ⋮ ; r_{N} ]_{N×1}

= [ ϕ_{1}^{T} ; ⋮ ; ϕ_{N}^{T} ] w + r = Φ^{T} w + r  (5.64)

where, as previously defined in Eq. (5.42),

ϕ_{n} = ϕ(x_{n}) = [ φ_{1}(x_{n}) ; ⋮ ; φ_{K}(x_{n}) ] , Φ = [ϕ_{1}, ⋯, ϕ_{N}]  (5.65)

As this nonlinear model ŷ = Φ^{T} w is in the same form as the linear model ŷ = X^{T} w, all previous discussion for the linear Bayesian regression is still valid for this nonlinear model, if X = [x_{1}, ⋯, x_{N}]_{(d+1)×N} is replaced by Φ = [ϕ_{1}, ⋯, ϕ_{N}]_{K×N}. We can get the covariance matrix and mean vector of the posterior p(w|D) in the K-dimensional space based on the observed data D = {X, y}:

Σ_{w/d} = (\frac{1}{σ_{r}^{2}} ΦΦ^{T} + Σ_{w}^{−1})^{−1}
m_{w/d} = \frac{1}{σ_{r}^{2}} Σ_{w/d}Φy = (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Φy  (5.66)

The function ŷ_{∗} = f(X_{∗}) of the multiple test points in X_{∗} = [x_{∗1}, ⋯, x_{∗M}]^{T} is a linear transformation of w:

ŷ_{∗} = f_{∗} = f(x_{∗}) = [ f_{1}(x_{∗}) ; ⋮ ; f_{M}(x_{∗}) ] = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{M}^{T} ]_{∗} w = [ϕ_{1}, ⋯, ϕ_{M}]^{T} w = Φ_{∗}^{T} w

(5.67)

and it also has a Gaussian joint distribution:

p(f_{∗}|X_{∗}, X, y) = N (Φ_{∗}^{T} m_{w/d}, Φ_{∗}^{T} Σ_{w/d}Φ_{∗})  (5.68)

with the mean and covariance:

E[f_{∗}|X_{∗}, X, y] = E[Φ_{∗}^{T} w] = Φ_{∗}^{T} E[w] = Φ_{∗}^{T} m_{w/d}
Cov[f_{∗}|X_{∗}, X, y] = E[f_{∗}f_{∗}^{T}] = E[Φ_{∗}^{T} ww^{T} Φ_{∗}]
= Φ_{∗}^{T} E[ww^{T}]Φ_{∗} = Φ_{∗}^{T} Σ_{w/d}Φ_{∗}  (5.69)

They can be reformulated to become:

E[f_{∗}|X_{∗}, X, y] = Φ_{∗}^{T} m_{w/d} = Φ_{∗}^{T} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} Φy
= Φ_{∗}^{T} (Σ_{w}Φ)(Σ_{w}Φ)^{−1} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1})^{−1} (Φ^{−1})^{−1}y
= Φ_{∗}^{T} Σ_{w}Φ [Φ^{−1} (ΦΦ^{T} + σ_{r}^{2}Σ_{w}^{−1}) Σ_{w}Φ]^{−1} y
= Φ_{∗}^{T} Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}y  (5.70)

and

Cov[f_{∗}|X_{∗}, X, y] = Φ_{∗}^{T} Σ_{w/d}Φ_{∗} = Φ_{∗}^{T} (\frac{1}{σ_{r}^{2}} ΦΦ^{T} + Σ_{w}^{−1})^{−1} Φ_{∗}
= Φ_{∗}^{T} [Σ_{w} − Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}Φ^{T} Σ_{w}] Φ_{∗}
= Φ_{∗}^{T} Σ_{w}Φ_{∗} − Φ_{∗}^{T} Σ_{w}Φ(Φ^{T} Σ_{w}Φ + σ_{r}^{2}I)^{−1}Φ^{T} Σ_{w}Φ_{∗}(5.71)

where we have used Woodbury identity in Section A.2.4.

We note that in these expressions the mapping function φ(x) only shows up in the quadratic form of Φ^{T} Σ_{w}Φ, Φ^{T} Σ_{w}Φ_{∗}, or Φ_{∗}^{T} Σ_{w}Φ_{∗}. This suggests that we may consider using kernel method (Section 14.2, by which some significant advantage may be gained (such as in the support-vector machines (Section 14). Specifically, if we replace the mn-th component ϕ^{T} (x_{m})Σ_{w}ϕ(x_{n}) of Φ^{T} Σ_{w}Φ by a kernel function only in terms of x_{m} and x_{n}:

ϕ^{T} (x_{m})Σ_{w}ϕ(x_{n}) = k(x_{m}, x_{n}) = k(x_{n}, x_{m})  (5.72)

then the mapping function φ(x) no longer needs to be explicitly specified. This idea is called the kernel trick, Now we have

Φ^{T} Σ_{w}Φ = [ ϕ_{1}^{T} ; ⋮ ; ϕ_{N} ] Σ_{w} [ϕ_{1}, ⋯, ϕ_{N}] = [ \begin{matrix}k(x_{1}, x_{1}) & ⋯ & k(x_{1}, x_{N}) \\ ⋮ & ... & ⋮ \\ k(x_{N}, x_{1}) & ⋯ & k(x_{N}, x_{N}) \end{matrix} ] = K

(5.73) and similarly we also have Φ^{T} Σ_{w}Φ_{∗} = K_{∗} and Φ_{∗}^{T} Σ_{w}Φ = K_{∗}^{T}. These kernel functions are symmetric, same as all covariance matrices. Now the mean and covariance of f_{∗} given above can be expressed as:

E[f_{∗}] = K_{∗}^{T} (K + σ_{r}^{2}I)^{−1} y
Cov[f_{∗}] = K_{∗∗} − K_{∗}^{T} (K + σ_{r}^{2}I)^{−1} K_{∗}  (5.74)

As a general summary of the method discussed above, we note that it is essentially different from the frequentist approach, in the sense that the estimated result produced is no longer a set specific values of some unknown variables in question (function values, parameters of regression models, etc.), but the probability distribution of these variables in terms of the expectation and covariance. Consequently, we can not only treat the expectation as the estimated values, but also use the variance as the confidence or certainty of such an estimation.

Below is a segment of Matlab code for estimating the mean and covariance of weight vector w by the Bayes linear regression. Here X and y are for X and y of the training data D, and Xstar, Phi and Phistar are for X_{∗}, Φ and Φ_{∗}, respectively.

    SigmaP=eye(d);   % prior variance of r in original space
    SigmaQ=eye(D);   % prior variance of r in feature space
    Sigmaw1=inv(X*X'/sigmaN^2+inv(SigmaP));
                     % covariance of w by Bayes regression
    Sigmaw2=inv(Phi0*Phi0'/sigmaN^2+inv(SigmaQ));
                     % cov of w with basis function
    w0=inv(X*X')*X*y % weight vector w by LS method
    w1=Sigmaw1*X*y/sigmaN^2 % mean of w by Bayes regression
    w2=Sigmaw2*Phi0*y/sigmaN^2;
                     % mean of w by Bayes regression with basis function
    y0=w0'*Xstar;    % y by least squares method
    y1=w1'*Xstar;    % y by Bayes regression
    y2=w2'*Phistar0; % y by Bayes regression with basis function
    for i=1:M
        sgm1(i)=sqrt(Xstar(:,i)'*Sigmaw1*Xstar(:,i));
                       % variance of estimated y
        sgm2(i)=sqrt(Phistar(:,i)'*Sigmaw2*Phistar(:,i));
    end

Example 5.11 Consider a 2-D linear function

f(x) = w^{T} x = [w_{0}, w_{1}] [ \begin{matrix}x_{0} \\ x_{1} \end{matrix} ] = w_{0}x_{0} + w_{1}x_{1} = 0.5 + 0.5 x_{1}  (5.75)

Figure 5.24 Prior and Posterior of Bayesian Regression

where we have assumed x_{0} = 1, w_{0} = −1/2 and w_{1} = 1/2, i.e., the function is actually a straight line f(x) = (−1 + x)/2 with intercept w_{0} = −1/2 and slope w_{1} = 1/2. The distribution of the additive noise (residual) is p(e) = N (0, σ_{r} = 0.8). The prior distributions of the weights w = [w_{0}, w_{1}]^{T} is assumed to be p(w) = N (0, I), and posterior p(w|D) can be obtained based on the observed data D = {X, y}, both shown in Fig. 5.24. We see that the posterior is concentrated around ŵ_{1} = −0.28 and ŵ_{2} = 0.39, the estimates of the true weights w_{1} = −0.5 and w_{2} = 0.5, based on the observed noisy data.

Various regression methods including both the least-squared method and the Bayesian linear regression methods, and in feature space spanned by a set of basis functions ϕ_{i}(x), (i = 1, ⋯, D = 5) based on each of the three functions given above are carried out with their results shown in Fig. 5.25.

The LS errors for these methods are listed below:

Method  LS Error w = [w_{1}, w_{2}]^{T}
Pseudo inverse 0.4730 [−0.28, 0.39]^{T}

Bayesian linear regression 0.4742 [−0.22, 0.38]^{T}

ϕ = exp(a||x − c||^{2})  0.6196
ϕ = ||x − c||  0.4429
ϕ = exp(−a||x − c||^{2})  0.4459

(5.76)

We see that the linear methods (the pseudo inverse and the Bayesian regression) produce similar estimated weights error, but smaller error can be achieved by the same method of Bayesian linear regression if proper nonlinear mapping is used (e.g., the last two mapping functions).

Problems

1. Develop Matlab code for both left and right pseudo inverse (Section A.6.1) of an m × n matrix A (m > n or m < n).

Figure 5.25 Bayesian Regression (Linear and Basis Function Based)

2. Solve the over-constrained linear equation system Ax = b and find the squared error, with the coefficient matrix

A = [ \begin{matrix}2 & 3 \\ 3 & −2 \\ 1 & 1 \end{matrix} ]

and each of the two different right-hand sides:

b_{1} = [ \begin{matrix}8 \\ −1 \\ 3 \end{matrix} ] , b_{1} = [ \begin{matrix}8 \\ 1 \\ 3 \end{matrix} ]

3. Develop Matlab code to solve ordinary least squares regression problems to find the d + 1 dimensional weight vector w = [w_{0} = b, w_{1}, ⋯, w_{d}]^{T}, and also other associated parameters including the sum of squares ESS, RSS, TSS, R^{2}, as well as the correlation coefficient ρ.

Apply your code to the following two data sets, find w and all parameters above, and then plot the regression curve y = w_{0} + w_{1}x together with the data points {(x_{i}, y_{i}) i = 1, ⋯, N}.

• 2-D dataset in/e176/programs/dataLS1.txt, containing two columns for (x, y) of each data point.

• 3-D dataset in /e176/programs/dataLS2.txt, containing three columns for (x_{1}, x_{2}, y) of each data point.

4. Implement ridge regression and apply it to a set of synthetic 2-D data (generated by your own code) including an independent variable x_{1} and another closely correlated variable x_{2} (a noisy version of the x_{1}, by, for example, x2=x1+a*rand(1,N)). Try different noise level by varying a see how ill-behaved the dataset is in terms of correlation coefficient ρ between the two variables, and how ill-conditioned the problem is in terms of how close to zero one of the eigenvalues of matrix XX^{T} is. Try different λ values of the ridge regression (including λ = 0 for regular linear regression) to see how it affects the stability of the resulting coefficients w_{0}, w_{1}, w_{2} and the accuracy in terms of R^{2}.

5. Implement the method of nonlinear regression based on each of four different basis functions considered in Section 5.3. Apply your code to solve the same 1-D regression problems given in Example 5.8. Compare results with different K values to see how the number of basis functions affects the results.

6. Develop Matlab code to implement the method of nonlinear regression based on each of the three basis functions, Gaussian, ReLU, and sigmoid considered in Section 5.3. Apply the code to solve the same 2-D regression problems given in Examples 5.9 and 5.10.

7. Develop Matlab code to implement the method of Bayesian regression based on different basis functions. Apply the code to the same 1-D problem in Example 5.11, and compare results with that of the LS method.

8. Apply the code developed in the previous problem to the same 2-D regression problem given in Examples 5.9 and 5.10.

6 Nonlinear Regression

6.1 Nonlinear Least Squares Regression

The goal of nonlinear least squares regression is to model the nonlinear relationship between the dependent variable y and independent variables in x = [x_{1}, ⋯, x_{d}]^{T}, based on the given training dataset D = {(x_{n}, y_{n}), n = 1, ⋯, N}, by a nonlinear regression function parameterized by θ = [θ_{1}, ⋯, θ_{M}]^{T}:

ŷ = f(x_{1}, ⋯, x_{d}, θ_{1}, ⋯, θ_{M}) = f(x, θ)  (6.1)

While the specific form of the nonlinear function is typically determined based on certain prior knowledge, we need to find the parameter θ for the model to fit the training dataset D, so that, its output ŷ matches the ground truth labeling y in the training set, i.e., ideally the residual r = y − ŷ below is zero:

r(θ) = [ r_{1} ; ⋮ ; r_{N} ] = [ y_{1} ; ⋮ ; y_{N} ] − [ f_{1}(θ) ; ⋮ ; f_{N}(θ) ] = y − ŷ(θ) = y − f(X, θ) = 0 (6.2)

where ŷ(x_{n}, θ) = f_{n}(θ) = f(x_{n}, θ) is the output generated by the regression function based on the nth data point x_{n}. We see that the regression problem is equivalent to solving a nonlinear equation system of N equations for the M unknown parameters in θ. As typically there are many more data samples than the unknown parameters, i.e., N ≫ M, the equation r = 0 is over-overconstrained without an exact solution. We need to find the the optimal parameter θ^{∗} that minimizes the sum of squared error (SSE) based on the residual r:

ε(θ) = \frac{1}{2}||r||^{2} = \frac{1}{2} ∑_{n=1}^{N} r_{n}^{2} = \frac{1}{2} ∑_{n=1}^{N}[y_{n} − f_{n}(θ)]^{2}  (6.3)

This optimization problem can be addressed by either the gradient descent methods or the Gauss-Newton method.

We first consider the gradient descent method based on the gradient of this error function:

g_{ε}(θ) = \frac{d}{dθ} ε(θ) = \frac{d}{dθ} (\frac{1}{2} ∑_{n=1}^{N} r_{n}^{2}) = ∑_{n=1}^{N} \frac{d r_{n}}{dθ} r_{n}

= − ∑_{n=1}^{N} \frac{d}{dθ} f_{n}(θ)[y_{n} − f_{n}(θ)] = −J_{f}^{T} r  (6.4)

where J_{f} is the N × M Jacobian matrix J_{f} of f(X, θ). We can also obtain the mth (m = 1, ⋯, M) component of the gradient vector as:

g_{m}(θ) = \frac{∂}{∂θ_{m}} ε(θ) = \frac{1}{2} \frac{∂}{∂θ_{m}} ||r||^{2} = \frac{1}{2} ∑_{n=1}^{N} \frac{∂}{∂θ_{m}} r_{n}^{2} = ∑_{n=1}^{N} \frac{∂r_{n}}{∂θ_{m}} r_{n}

= − ∑_{n=1}^{N} \frac{∂f_{n}(θ)}{∂θ_{m}}[y_{n} − f_{n}(θ)] = − ∑_{n=1}^{N} J_{nm} [y_{n} − f_{n}(θ)] = − ∑_{n=1}^{N} J_{nm} r_{n} (6.5)

where J_{nm} = ∂ f_{n}(θ)/∂θ_{m} is the component in the nth row and mth column of J_{f}, which is simply the negative version of Jacobian of r = y − f(X, θ):

J_{r}(θ) = \frac{d}{dθ} r(θ) = \frac{d}{dθ} [y − f(X, θ)] = −\frac{d}{dθ} f(X, θ) = −J_{f}(θ)  (6.6)

The optimal parameter θ that minimizes ε(θ) can now be found iteratively by the gradient descent method

θ_{n+1} = θ_{n} − δ_{n}g_{ε}(θ_{n}) = θ_{n} + δ_{n}J_{f}^{T} r  (6.7)

Setting the value for the step size δ_{n} properly may be tricky, as it depends on the specific landscape of the error hypersurface ε(θ). If δ_{n} is too large, a minimum may be skipped over, but if δ_{n} is too large, the convergence of the iteration may be too slow.

This gradient method is also called batch gradient descent (BGD), as the gradient in Eq. (6.4) is calculated based on the entire batch of all N data points in the dataset D in each iteration. When the size of the dataset is large, BGD may be slow due to the high computational complexity. In this case the method of stochastic gradient descent (SGD) can be used, which approximates the gradient based on only one of the N data points uniformly randomly chosen from the dataset in each iteration, i.e., the summation in Eq. (6.4) contains only one term corresponding to the chosen sample x_{n}:

g_{n}(θ) = \frac{d}{dθ} \frac{1}{2}r_{n}^{2} = r_{n} \frac{d r_{n}}{dθ}  (6.8)

The expectation of g_{n} based on one data point is the same as the true gradient (scaled by 1/N) based on all N data points in the dataset

E[g_{n}(θ)] = ∑_{n=1}^{N} P (x_{n})g_{n}(θ) = ∑_{n=1}^{N} \frac{1}{N} g_{n}(θ) = \frac{1}{N} ∑_{n=1}^{N} \frac{dr_{n}}{dθ} r_{n} = \frac{1}{N} g(θ) (6.9)

As the data are typically noisy, the SGD gradient based on only one data point may be in a direction very different from the true gradient based on the mean of all N data points, and the resulting search path in the M-D space may be in a very jagged zig-zag shape, and the iteration may converge slowly. However, as the computational complexity in each iteration is much reduced, we can afford to carry out a large number of iterations with an overall search path generally coinciding with that of the BGD method.

Some tradeoff between BGD and SGD can be made so that the gradient is estimated based on more than one but less than all N data points in the dataset, the resulting method, called mini-batch gradient descent, can take advantage of both methods. The methods of stochastic and mini-batch gradient descent are widely used in machine learning, such as in the method of artificial neural networks, which can be treated as a nonlinear regression problem, considered in Part V.

We then consider the Gauss-Newton method to find the optimal parameter θ^{∗} of the nonlinear regression function f(x, θ) by directly solving the nonlinear over-determined equation system r(θ) = y − f(X, θ) = 0 in Eq. (6.2), based on the iteration in Eq. (1.77) in Chapter 1:

θ_{n+1} = θ_{n} − J_{r}^{−} r = θ_{n} − (J_{r}^{T} J_{r})^{−1}J_{r}^{T} r = θ_{n} + (J_{f}^{T} J_{f})^{−1}J_{f}^{T} r  (6.10)

The last equality is due to the fact that J_{r} = −J_{f}. Here J_{f}^{T} J_{f} in the pseudoinverse is actually an approximation of the Hessian H_{ε} of the error function ε(θ), as given in Eq. (2.20) in Chapter 2, and −J_{f}^{T} r = g_{ε}(θ) is the gradient of the error function given in Eq. (6.4). So the iteration above is essentially:

θ_{n+1} = θ_{n} − J_{r}^{−} r ≈ θ_{n} − H_{ε}^{−1}(θ_{n})g_{ε}(θ_{n})  (6.11)

same as the Newton-Raphson method based on the second order Hessian as well as the first order gradient. We therefore see that the Gauss-Newton method is actually for minimizing the sum of squares error is essentially the same as the Newton-Raphson method, with the only difference that the Hessian H_{ε} ≈ J_{f}^{T} J_{f} as the second order derivative of the error function ε(θ) is approximated by the Jacobian J_{f}θ as the first order derivative of the regression function f(x, θ), making the implementation computationally easier.

In the special case of linear regression function f(w) = x^{T} w, we have:

ŷ = f(X, w) = [ f(x_{1}, w) ; ⋮ ; f(x_{N}, w) ] = [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] w = [x_{1}, ⋯, x_{N}]^{T} w = X^{T} w (6.12)

where w = [w_{0}, w_{1}, ⋯, w_{d}]^{T} and x_{i} = [1, x_{i1}, ⋯, x_{id}]^{T} are both augmented d + 1 dimensional vectors and X = [x_{1}, ⋯, x_{N}]. The Taylor series is

f(X, w + Δw) = X^{T} (w + Δw) = X^{T} w + X^{T} Δw  (6.13)

and the Jacobian is

J_{f}(w) = X^{T}  (6.14)

Now Eq. (6.10) becomes

w_{n+1} = w_{n} + (X^{T})^{−}(y − X^{T} w_{n}) = w_{n} + (XX^{T})^{−1}X(y − X^{T} w_{n})
= w_{n} + (XX^{T})^{−1}Xy − (XX^{T})^{−1}(XX^{T})w_{n}
= (XX^{T})^{−1}Xy = (X^{T})^{−}y  (6.15)

i.e., the optimal parameter w^{∗} can be found in a single step without iteration, same as that in Eq. (5.10).

The Matlab code for the essential parts of the algorithm is listed below, where data(:,1) and data(:,2) are the observed data containing N samples x_{1}, ⋯, x_{N} in the first column and their corresponding function values y_{1}, ⋯, y_{N} in the second column. In the code segment below, the regression function (e.g., an exponential model) is symbolically defined, and then a function NLregression is called to estimate the function parameters:

sym x;                      % symbolic variables
a=sym('a',[3 1]);           % symbolic parameters
f=@(x,a)a(1)*exp(a(2)*x)+a(3)|    % symbolic function
[a er]=NLregression(data(:,1),data(:,2),f,3);
                        % call regression

The code below estimates the M parameters of the regression function based on Newton’s method.

function [a er]=NLregression(x,y,f,M)   % nonlinear regression
N=length(x);            % number of data samples
a=sym('a',[M 1]);       % M symbolic parameters
F=[];
for i=1:N               % for N given points
    F=[F; f(x(i),a)];   % evaluate function at each point
end
J=jacobian(F,a);        % symbolic Jacobian
J=matlabFunction(J);    % convert to a true function
F=matlabFunction(F);
a1=ones(M,1);           % initial guesses for M parameters
n=0;                    % iteration index
da=1;
while da>tol            % terminate if no progress made
    n=n+1;
    a0=a1;              % update parameters
    J0=J(a0(1),a0(2));  % evaluate Jacobian based on a0
    Jinv=pinv(J0);      % pseudo inverse of Jacobian
    r=y-F(a0(1),a0(2))  % residual
    a1=a0+Jinv*r;       % Newton iteration
    er=norm(r);         % residual error
    da=norm(a0-a1);     % progress

        end
    end

where tol is a small value for tolerance, such as 10^{−6}.

Example 6.1 Consider three sets of data (shown as the solid dots in the three columns in the plots in Fig. 6.1) are fitted by three different models (shown as the open circles in the plots):

1. Linear model: y = f_{1}(x, a, b) = ax + b

2. quadratic model: y = f_{2}(x, a, b, c) = ax^{2} + bx + c

3. Exponential model: y = f_{3}(x, a, b, c) = a e^{bx} + c

These are the parameters for the three models to fit each of the three datasets:

P arameters :  a  b  c  error
Model 1 :  30.337 51.652  7.4294
Dataset 1  Model 2 :  0.076 30.034  51.580 7.4223
Model 3 : 5933.214 0.005 −5881.634 7.4221
Model 1 :  24.519 20.089  21.0362
Dataset 2  Model 2 :  4.459 6.625  15.855 8.4559
Model 3 :  29.791 0.346  −10.202 9.7402
Model 1 :  32.608 22.008  53.0724
Dataset 3  Model 2 :  10.391 −9.089  12.140 28.3211
Model 3 :  2.197 0.885  27.241 8.3393

We see that the first dataset can be fitted by all three models equally well with similar error (when a ≈ 0 in model 2, it is approximately a linear functions), the second dataset can be well fitted by either the second or the third model, while the third dataset can only be fitted by the third exponential model.

Example 6.2 Based on the following three functions with four parameters a = 1, b = 2, c = 3, d = 4, three sets of synthetic 2-D data points are generated with some random noise added, shown in Fig. 6.2 as the dots in the three rows.

f_{1}(x) = ax_{1} + bx_{2} − c
f_{2}(x) = exp(ax_{1} + bx_{2}) − cx_{1}x_{2} + d
f_{3}(x) = ax_{1}^{2} − bx_{1}x_{2} + cx_{2}^{2} + d  (6.16)

Then each of three datasets are fitted by the three functions as models while the optimal parameters are found by the Gauss-Newton method.

Figure 6.1 Nonlinear Regression with Multiple Datasets and Models (1-D)

\begin{aligned}& P arameters : & a & b & c & d & error \\ Dataset 1 & Model 1 : & 0.98 & 1.97 & 3.00 & & 4.13 \\ & Model 2 : & 0.70 & 1.30 & 1.21 & −4.42 & 6.87 \\ & Model 3 : & −0.19 & 0.73 & −0.12 & −3.01 & 18.03 \\ Dataset 2 & Model 1 : & 1.84 & 3.40 & −6.09 & & 14.70 \\ & Model 2 : & 0.99 & 2.01 & 2.97 & 3.98 & 3.75 \\ & Model 3 : & 0.37 & 1.13 & 2.66 & 4.87 & 32.45 \\ Dataset 3 & Model 1 : & −0.19 & 0.30 & −5.34 & & 16.82 \\ & Model 2 : & −0.29 & 0.23 & 2.00 & 4.30 & 13.51 \\ & Model 3 : & 0.94 & 2.05 & 2.99 & 4.02 & 4.09 \end{aligned}

The Levenberg-Marquardt algorithm (LMA) or damped least-squares method is a method widely used to minimize a sum of squares error ε as in Eq. (6.3). The LMA is an interpolation or tradeoff between the the gradient descent method based on the first order derivative of the error function to be minimized, and the Gauss-Newton method based on both the first and second order derivatives of the error function, both discussed above. As the LMA can take advantage of both methods and is therefore more robust. Even if the initial guess is far from the solution corresponding to the minimum of the objective function, the iteration can still converge toward the solution.

Figure 6.2 Nonlinear Regression with Multiple Datasets and Models (2-D)

Specifically, the LMA modifies Eq. (6.10)

θ_{n+1} = θ_{n} + J_{f}^{−}(θ_{n})[y − f(X, θ_{n})] = θ_{n} + (J_{n}^{T} J_{n})^{−1}J_{n}^{T} [y − f(X, θ_{n})] (6.17)

to include an extra term proportional to the identity matrix I:

θ_{n+1} = θ_{n} + (J_{n}^{T} J_{n} + λI)^{−1}J_{n}^{T} [y − f(X, θ_{n})]  (6.18)

where λ ≥ 0 is the non-negative damping factor, which is to be adjusted at each iteration. We recognize that this is actually similar to Eq. (5.34) for ridge regression.

In each iterative step, if λ is small and the error function ε(θ_{n}) decreases quickly, indicating the Gauss-Newton is effective in the sense that the error function is well modeled as a quadratic function by the first and second order terms of the Taylor series, we will keep λ small. On the other hand, if ε decreases slowly, indicating the error function may not be well modeled as a quadratic function, due possibly to the fact that the current θ_{n} is far away from the minimum, then we let λ take a large value, so that the term J_{n}^{T} J_{n} becomes insignificant, and Eq. (6.18) approaches

θ_{n+1} = θ_{n} + λJ_{n}^{T} (y − f(θ)) = θ_{n} − αg_{ε}(θ)  (6.19)

Now θ_{n} is updated along the negative direction of the gradient in Eq. (6.4):

g_{ε}(θ) ∝ J^{T} (f(θ) − y)  (6.20)

with some step size α.

By adjusting the parameter λ, the LMA can switch between either of the two methods for the same purpose of minimizing ε(θ). When the value of λ is small, the iteration is dominated by the Gauss-Newton method that approximate the error function by a quadratic function based on the second order derivatives. If the error reduces slowly, indicating the error function cannot be approximated by a quadratic function, a greater value of λ is used to switch to the gradient descent method.

6.2 Parameter Estimation by Natural Gradient Descent

In Section 4.1, we considered the MLE and MAP methods for estimating the parameter θ of a probability model p(x|θ) by maximizing the likelihood function L(θ|x) or the posterior p(θ|x). In the previous section we also considered how the gradient method is used to estimate the model parameter θ by minimizing the squared error. Obviously we can also use the gradient method in MLE to maximize the likelihood. However, unlike the parameter θ = [θ_{1}, ⋯, θ_{m}]^{T} which is treated as a vector in an m-D Euclidean space spanned by a set of orthonormal basis vectors, the family of distributions p(x|θ) corresponding to all possible parameters θ resides in a distribution space, an m-dimensional Riemannian manifold or space, in which no orthogonal basis is defined, and methods based on concepts such as inner product and Euclidean distance are no longer valid.

In this case, the difference between any two distributions p(x|θ) and p(x|θ′) can be measured by their KL-divergence KL(p(x|θ)||p_{p}(x|θ + Δθ)), and the gradient method can still be used to find iteratively the optimal parameter that maximize the likelihood L(θ), similar to the steps that lead to Eq. (3.32). Specifically, we desired to find the optimal Δθ that maximally increases the likelihood, which is also subject to the constraint ||Δθ||^{2} < C:

Δθ^{∗} = arg max_{||Δθ||^{2}<C} L(θ + Δθ)  (6.21)

where ||Δθ||^{2} is defined as Eq. (B.275):

||Δθ||^{2} = \frac{1}{2}Δθ^{T} F(θ)Δθ = KL(p(x|θ)||p_{p}(x|θ + Δθ)) < C  (6.22)

To solve this constrained optimization problem we set the derivative of the Lagrangian to zero

\frac{d}{d(Δθ)} [L(θ + Δθ) + μ (\frac{1}{2}Δθ^{T} F(θ)Δθ − C)] = g_{L}(θ) + μF(θ)Δθ = 0

(6.23)

Solving this equation we get the optimal increment:

Δθ^{∗} = −\frac{1}{μ}F^{−1}(θ)g_{L}(θ)  (6.24)

and the optimal parameter can be found iteratively using this method of natural gradient descent (Eq. (B.3.5):

θ_{n+1} = θ_{n} + Δθ_{n} = θ_{n} + λF^{−1}g_{L}(θ_{n}) = θ_{n} + λg̃_{L}(θ_{n})  (6.25)

where λ = 1/μ is the step size, and g̃_{L}(θ_{n}) is the natural gradient:

g̃_{L}(θ_{n}) = F^{−1}g_{L}(θ_{n})  (6.26)

Comparing this iteration with that in Eq. (2.15), we see that the natural gradient method is similar to the Newton-Raphson method in the sense that it is based on not only the gradient g_{J}(θ) for the local first order variation of the likelihood function L(θ) in the distribution space, but also the Fisher information F(θ) for the more global second order curvature in the space, by which the gradient direction is modified so that the iterative search process in the space can be carried out more efficiently.

Problems

1. Develop Matlab code to implement the nonlinear least-squares regression algorithm considered in Section 6, and apply your code to resolve the 1-D nonlinear regression problem in Example 6.1 The datasets can be downloaded from here....

2. Apply your code to resolve the 2-D nonlinear regression problem such as that in Example 6.2. Three data sets can be found below to find the optimal model parameters (a, b, c, d) for your models.

• dataset 1 in /e176/homework e190/data3.txt

• dataset 2 in /e176/homework e190/data4.txt

• dataset 3 in /e176/homework e190/data5.txt

You can read these data into your Matlab code by data=load('data3.txt'). Each data file is organized in three columns, the first two are for the two independent variables x_{1} and x_{2} representing a set of data points in a 2-D space, while the third one is for the function values f(x_{1}, x_{2}) corresponding to these points.

7 Logistic and Softmax Regression

7.1 Logistic Regression and Binary Classification

As mentioned previously, regression methods can be used as a binary classifier, when the regression function f(x, θ) is thresholded by some constant c, which is assumed to be zero in the following without loss of generality. Once the model parameter θ is obtained based on the training set D = {X, y}, every point x in the d-dimensional feature space can be classified into either of two classes C_{−} and C_{+}:

if f(x) \{ \begin{matrix}< c \\ > c \end{matrix}  then x ∈ \{ \begin{matrix}C_{−} \\ C_{+} \end{matrix}  (7.1)

When c = 0, the regression function f(x) is thresholded by a unit step function u(x) (u(x) = 1 if x ≥ 0 but 0 otherwise), and the corresponding equation f(x) = 0 represents a hypersurface in the d-dimensional space (a point if d = 1, a curve if d = 2, a surface or hypersurface if d ≥ 3), by which the space is partitioned into two regions. In this case, f(x) is called the decision function and the surface decision boundary. Now any point in the space is classified into either of the two classes, depending on whether it is on the positive side of the decision boundary (f(x) > 0) or the negative side (f(x) < 0).

This simple binary classifier suffers from the drawback that the classification result is binary and deterministic. It provides no additional information such as how reliable the result is, i.e., how confident we can be regarding the result. Specifically, such a classifier simply classifies all points on one side of the boundary into the same class, without considering how far they are from the decision boundary, while one would naturally feel more confident about the classification of points far away from the boundary, than those close to it. In other words, this binary classifier may tend to overfit as it is sensitive to noisy data points close to the boundary.

This problem can be addressed by the method of logistic regression, which can provide the probability for any given data point to be classified to either of the two classes, as a quantitative measurement of the confidence for the classification. Based on such a probabilistic result, one can not only make a binary decision, but also gain the extra information in terms of the reliability of the decision.

The logistic regression is based on the same linear regression function f(x) =

Figure 7.1 Unit Step vs. Sigmoid

w^{T} x. However, different from linear regression, in logistic regression, the linear function is not hard-thresholded by the unit step function to become either 0 or 1; instead, it is soft-thresholded by a sigmoid function σ(f(x)) = σ(w^{T} x) to produce a value between 0 and 1, as shown in Fig. 7.1, which can then be interpreted as the probability p(x ∈ C_{+}) for x to belong to C_{+}, and the probability for x to belong to C_{−} is simply p(x ∈ C_{−}) = 1 − p(x ∈ C_{+}).

Similar to linear and nonlinear regression methods considered before, the goal of logistic regression is also to find the optimal model parameter w for the regression function f(x, w) to fit the training set optimally. But different from linear and nonlinear regression based on the least squares (LS) method that minimizes the squared error in Eq. (6.3), logistic regression is based on maximum likelihood (ML) estimation, which maximizes the likelihood of the model parameter, as discussed below.

The sigmoid function can be either the error function (cumulative Gaussian, erf):

ϕ(x) = ∫_{−∞}^{x} N (u|0, 1) du = \frac{1}{\sqrt{2π}} ∫_{−∞}^{x} exp (−\frac{1}{2N} u^{2}) du  (7.2)

or the logistic function:

σ(x) = \frac{1}{1 + e^{−x}} = \frac{e^{x}}{1 + e^{x}} = \{ \begin{matrix}1 & x → ∞ \\ 1/2 & x = 0 \\ 0 & x → −∞ \end{matrix}  (7.3)

In the following, we will only consider the logistic function for consistency and simplicity. Here is a set of properties of the logistic function:

σ(−x) = \frac{1}{1 + e^{x}} = 1 − \frac{1}{1 + e^{−x}} = 1 − σ(x)

\frac{d}{dx}σ(x) = \frac{e^{−x}}{(1 + e^{−x})^{2}} = σ(x)σ(−x) = σ(x)(1 − σ(x))

x = ln (\frac{σ(x)}{1 − σ(x)}) , e^{x} = \frac{σ(x)}{1 − σ(x)}  (7.4)

Moreover, a parameter w can be included in the logistic function to control the softness or smoothness of the transition between 0 and 1, as shown in Fig. 7.2.

Figure 7.2 Sigmoid with Different Parameter w

σ(wx) = \{ \begin{matrix}1/2 & w = 0 \\ 1/(1 + e^{−wx}) & w > 0 \\ u(x) & w → ∞ \end{matrix}  (7.5)

We see that the argument x is thresholded by the logistic function with variable softness controlled by parameter w, between the two extreme cases of σ(w x) = 1/2 (no thresholding) when w = 0, and σ(w x) = u(x) when w → ∞ (hard thresholding).

Now the linear function f(x) = w^{T} x ∈ (−∞, ∞) is mapped by the sigmoid function σ(f(x)) to a real value between 0 and 1, which is interpreted as the conditional probability for output ŷ = 1 given data point x and model parameter w:

p(ŷ = 1|x, w) = σ(w^{T} x) = \frac{1}{1 + e^{−w^{T} x}} ∈ (0, 1)  (7.6)

Comparing this with Eq. (7.5) for w > 0, we note that the weight vector w controls the smoothness of the thresholoding.

This probability p(ŷ = 1|x, w) is also the probability p(x ∈ C_{+}) for x ∈ C_{+}, and the probability p(x ∈ C_{−}) is:

p(ŷ = 0|x, w) = 1−p(y = 1|x, w) = 1−σ(w^{T} x) = σ(−w^{T} x) = \frac{1}{1 + e^{w^{T} x}} (7.7)

Now any given x can be classified into either class C_{+} or class C_{−} depending on if σ(w^{T} x) > 1/2. Specifically, consider the following cases:

• If f(x) = w^{T} x ≫ 0, i.e., x is on the positive side far away from the plane f(x) = 0, then the probability p(x ∈ C_{+}) = σ(f(x)) is close to 1;

• If f(x) = w^{T} x ≪ 0, i.e., x is on the negative side far away from the plane

Figure 7.3 Convert Regression Function to Probability

f(x) = 0, then the probability p(x ∈ C_{+}) = σ(f(x)) is close to 0, and p(x ∈ C_{−}) = 1 − p(x ∈ C_{+}) is close to 1;

• If f(x) ≈ 0, then both p(x ∈ C_{+}) = σ(f(x)) and p(x ∈ C_{−}) = 1 − σ(f(x)) are approximately 1/2, i.e., if a point x is very close to the plane f(x) = 0, then the probabilities for it to belong to either C_{+} or C_{−} are about the same.

We note that the weight vector w plays the same role as w in Eq. (7.5) (w > 0), i.e., a larger value of ||w|| results in a sharper thresholding by the sigmoid function σ(w^{T} x). In other words, w determines not only how the classification is carried out based on the decision boundary, but also whether the classifier over and underfits, as further illustrated in the two extreme cases:

• ||w|| → ∞: the value of the sigmoid function σ(w^{T} x) is approximately binary, close to either 0 or 1, based on which the feature space is hard-divided into two regions each for one of the two classes. As such a binary classification is dictated mostly by a small number of data points close to the boundary between the two regions, it is prone to noise contained in these data points and tends to overfit.

• ||w|| → 0: the sigmoid function approaches a constant σ(w^{T} x) = 1/2, totally insensitive to the locations of the data points, i.e., the probability for any data point to belong to either of the two classes is always the same. As such a classifier is insensitive to the valid variations in the data, it tends to underfit.

We therefore see that by adjusting ||w|| and thereby controlling the smoothness of the thresholding, we can make a proper tradeoff between overfitting and underfitting.

We label the each sample x_{n} in the training set X = [x_{1}, ⋯, x_{N}] by either y_{n} = 1 if x_{n} ∈ C_{+}, or y_{n} = 0 if x_{n} ∈ C_{−}. Then the two probabilities in Eqs. (7.6) and (7.7) can be combined into the following conditional probability for the model output ŷ to match the ground truth labeling y_{n}:

p(ŷ = y_{n}|x_{n}, w) = P (ŷ = 1|x_{n}, w)^{y_{n}}p(ŷ = 0|x_{n}, w)^{1−y_{n}}
= \{ \begin{matrix}p(ŷ = 1|x_{n}, w) & if y_{n} = 1 \\ p(ŷ = 0|x_{n}, w) & if y_{n} = 0 \end{matrix}  (7.8)

We can further get the likelihood function of w given all N samples in D = {X, y} (assumed i.i.d.):

L(w|D) ∝ p(D|w) = p(y|X, w) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, w)

= ∏_{n=1}^{N} p(ŷ = 1|x_{n})^{y_{n}}p(ŷ = 0|x_{n})^{1−y_{n}} = ∏_{n=1}^{N} σ(w^{T} x_{n})^{y_{n}}(1 − σ(w^{T} x_{n}))^{1−y_{n}} (7.9)

The optimal model parameter w can be found by either of the two methods below:

• Maximum likelihood estimateion (MLE)

The optimal model parameter w is defined as the one that maximizes the likelihood function L(w|D), or, equivalently, that minimizes the negative log likelihood as the objective function:

J(w) = −l(w|D) = − log L(w|D)
= − ∑_{n=1}^{N} [y_{n} log σ(w^{T} x_{n}) + (1 − y_{n}) log(1 − σ(w^{T} x_{n}))] (7.10)

This is the general MLE method that is widely used to obtain the most probable estimation of certain parameter of a model based on the given data.

• Maximum a posteriori (MAP)

The optimal parameter w can also be defined as the one that maximizes the posterior probability:

p(w|D) = p(w|X, y) = \frac{p(y|X, w) p(w)}{p(y|X)} ∝ p(y|X, w) p(w)  (7.11)

where p(y|X, w) = L(w|D) is the likelihood given in Eq. (7.9), p(w) is the prior probability of w before observing the training data D, and the denominator p(yX) is dropped as it is independent of variable w.

Based on the assumption that all components of w are independent of each other and they have zero mean and the same variance, we let the prior probability be:

p(w) = N (0, σ^{2}I) = \frac{1}{(2πσ^{2})^{d/2}} exp (−\frac{||w||^{2}}{2σ^{2}} ) ∝ exp (−\frac{||w||^{2}}{2σ^{2}} ) (7.12)

where the normalizing factor 1/(2πσ^{2})^{d/2} independent of w can be dropped. The assumed zero mean is due to the desire that the values of all weights in w should be around zero instead of large values on either positive or negative side, to avoid overfitting, as discussed below.

Now the posterior can be written as

p(w|D) ∝ p(y|X, w) p(w) = ∏_{n=1}^{N} p(ŷ_{n} = y_{n}|x_{n}, w) N (0, Σ_{w})

∝ ∏_{n=1}^{N} σ(w^{T} x_{n})^{y_{n}}(1 − σ(w^{T} x_{n}))^{1−y_{n}} exp (−\frac{||w||^{2}}{2σ^{2}} )(7.13)

The MAP method finds the optimal model parameter w that maximizes this posterior p(w|D) or equivalently minimizes the negative log posterior as the objective function:

J(w) = − log p(w|D) = − log p(y|X, w) − (−\frac{||w||^{2}}{2σ^{2}} )

= − ∑_{n=1}^{N} [y_{n} log σ(w^{T} x) + (1 − y_{n}) log(1 − σ(w^{T} x))] + \frac{λ}{2}||w||^{2}(7.14)

where λ is a hyperparameter defined as λ = 1/σ^{2}. Comparing this expression with Eq. (5.32) for ridge regression, we note that the second term can be treated as a penalty term that discourages large values of w to avoid overfitting. When J(w) is minimized, a larger value of λ results in smaller values for w, and the sigmoid function will be more smooth for a softer thresholding, and the classification is less affected by the possible noisy outliers close to the decision boundary. By adjusting λ, we can make a proper tradeoff between overfitting and underfitting.

In particular, if σ → ∞ and λ → 0, then exp(−λ||w||^{2}/2) → 1, i.e., the prior p(w) approaches a uniform distribution (all values of w are equally likely), then the penalty term can be removed, and Eq. (7.14) becomes the same as Eq. (7.10), i.e., the MAP solution that maximizes the posterior p(w|D) becomes the same as the MLE solution that maximizes the likelihood L(w|D).

Here we consider the MAP method which contains an extra term ||w||^{2}//2σ^{2}) and is therefore more general. We first find the gradient of objective function:

g_{J}(w) = \frac{d}{dw} J(w) = \frac{d}{dw} (− log p(y|X, w) + \frac{λ}{2}||w||^{2})
= − ∑_{n=1}^{N} \frac{d}{dw} [y_{n} log σ(w^{T} x_{n}) + (1 − y_{n}) log(1 − σ(w^{T} x_{n}))] + λw

= − ∑_{n=1}^{N} [\frac{y_{n}}{σ(w^{T} x_{n})} − \frac{1 − y_{n}}{1 − σ(w^{T} x_{n})}] σ(w^{T} x_{n})(1 − σ(w^{T} x_{n})) x_{n} + λw

= ∑_{n=1}^{N} [σ(w^{T} x_{n}) − y_{n}] x_{n} + λw = [x_{1}, ⋯, x_{N}] [ σ(w^{T} x_{1}) − y_{1} ; ⋮ ; σ(w^{T} x_{N}) − y_{N} ] + λw
= Xr + λw  (7.15)

where we have used the properties of the logistic function in Eq. (7.4), and r(w) is the residual defined as

r = r(w) = [ r_{1}(w) ; ⋮ ; r_{N}(w) ] = [ σ(w^{T} x_{1}) − y_{1} ; ⋮ ; σ(w^{T} x_{N}) − y_{N} ]  (7.16)

containing all N residuals r_{n} = σ(w^{T} x_{n}) − y_{n} (n = 1, ⋯, N), the difference between the model prediction 0 < σ(w^{T} x_{n}) < 1 and the ground truth labeling y_{n} ∈ {0, 1}.

Based on g_{J}(w) given in Eq. (7.15), we can find the optimal w that minimizes this objective function by the either the gradient descent method or Newton’s method that depends on the second order Hessian matrix H_{J}(w) in addition to the first order gradient.

The Matlab code segment below is for the gradient descent method. Here X contains the N samples x_{1}, ⋯, x_{N} nd y contains the corresponding a binary labeling:

    X=[ones(1,n); X];      % augmented X with x_0=1
    w=ones(m+1,1);         % initialize weight vector
    g=X*(sgm(w,X)-y)';     % gradient of log posterior
    tol=10^(-6);           % tolerance
    delta=0.1;             % step size
    while norm(g)>tol      % terminate when g is small enough
        w=w-delta*g;       % update weight by gradient descent
        g=X*(sgm(w,X)-y)'; % update gradient
    end

where

    function s=sgm(w,X)
        s=1./(1+exp(-w'*X));

    end

We can also use Newton’s method if the second order Hessian matrix is available:

H_{J}(w) = \frac{d}{dw} g_{J}(w) = \frac{d}{dw} (Xr(w) + λ||w||) = X\frac{d}{dw} r(w) + λ

= X\frac{d}{dw} [ r_{1}(w) ; ⋮ ; r_{N}(w) ] + λ = XJ_{r}(w) + λ  (7.17)

where J_{r}(w) = dr(w)/dw is the N × (d + 1) Jacobian matrix of the vector function r(w), of which the nth row is

\frac{dr_{n}(w)}{dw} = \frac{d}{dw} [σ(w^{T} x_{n}) − y_{n}] = σ(w^{T} x_{n})[1 − σ(w^{T} x_{n})]x_{n} = z_{n}x_{n} (7.18)

where we have defined z_{n} = σ(w^{T} x_{n})(1 − σ(w^{T} x_{n})), which is always greater than zero as 0 < σ(w^{T} x_{n}) < 1. Now the Jacobian can be written as

J_{r}(w) = [ dr_{1}(w)/dw ; ⋮ ; dr_{N}(w)/dw ] = [ z_{1}x_{1} ; ⋮ ; z_{N}x_{N} ] = [ \begin{matrix}z_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & z_{N} \end{matrix} ] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = ZX^{T}

(7.19)

where Z = diag(z_{1}, ⋯, z_{N}) is a positive definite diagonal matrix (as z_{n} > 0). Substituting J_{r} back into Eq. (7.17), we get

H_{J} = XJ_{r} + λ = XZX^{T} + λ  (7.20)

Since Z is positive definite, the following quadratic function is also positive, i.e., for any y, we have

y^{T} H_{J}y = y^{T} XZX^{T} y + λy^{T} y = (X^{T} y)^{T} Z(X^{T} y) + λ||y|| > 0  (7.21)

indicating H_{J} is also positive definite, and the objective function J(w) when approximated by the first three terms of its Taylor series is has a global minimum.

Having found H_{J}(w) as well as g_{l} = J(w) of the objective function J(w), we can solve the minimization problem also by Newton’s method:

w_{n+1} = w_{n} − H_{J}(w_{n})^{−1}g_{J}(w_{n}) = w_{n} − (XZX^{T})^{−1}Xr  (7.22)

The following code segment estimates weight vector w by Newton method:

    X=[ones(1,n); X];          % augmented X with x_0=1
    w=ones(m+1,1);             % initialize weight vector
    g=X*(sgm(w,X)-y)';         % gradient of log posterior
    H=X*diag(s.*(1-s))*X';     % Hessian of log posterior
    tol=10^(-6)
    while norm(g)>tol          % terminate when g is small enough
        w=w-inv(H)*g;          % update weight by Newton-Raphson

        s=sgm(w,X);            % update sigma
        g=X*(s-y)';            % update gradient
        H=X*diag(s.*(1-s))*X'; % update Hessian
    end

Once the model parameter w is available, an unlabeled test sample x_{∗} of unknown class can be classified into either of the two classes C_{1} and C_{0}, based on p(y_{∗} = 1|x_{∗}):

if p(y_{∗} = 1|x_{∗}) = σ(w^{T} x_{∗}) \{ \begin{matrix}> 1/2 \\ < 1/2 \end{matrix} , then x_{∗} ∈ \{ \begin{matrix}C_{1} \\ C_{0} \end{matrix}  (7.23)

In summary, we make a comparison between the linear and logistic regression methods, both can be used for binary classification. In linear regression, the function value f(x) = w^{T} x is hard-thresholded by the unit step function, so that any data point x is classified to class C_{+} if w^{T} x > 0, or class C_{−} if w^{T} x < 0 in a deterministic fashion (with probability p(x ∈ C_{+}) ∈ {0, 1}. On the other hand, in logistic regression, the linear function f(x) = w^{T} x is soft-thresholded by the sigmoid function to become p(x ∈ C_{+}) = σ(f(x)) ∈ (0, 1), treated as the probability for x to belong to class C_{+}. Although the same linear model f(x) = w^{T} x is used in both cases, the two methods obtain the model parameter w by minimizing very different objective functions (Eqs. (5.8) and (7.14)).

7.2 Softmax Regression for Multiclass Classification

In a multiclass classification problem, an unlabeled data point x is to be classified into one of K > 2 classes {C_{1}, ⋯, C_{K}}, based on the training set D = {(x_{n}, y_{n}), n = 1, ⋯, N}, where y_{n} ∈ {1, ⋯, K} is an integer indicating x_{n} ∈ C_{y_{n}}

Any binary classifier, such as the logistic regression considered above, can be used to solve such a multiclass classification problem in either of the following two ways:

• One-Vs-One (1V1): construct K(K − 1)/2 binary classifiers for every pair of the K classes based on the training set {(C_{i}, C_{j}) i = 2, ⋯, K, j = 1, ⋯, i−1}, then an unlabeled sample x is classified to one of the K classes with maximum votes out of this many binary classifications.

• One-Vs-Rest (1VR). Convert a K-class problem into K binary problems by regrouping the K classes into two classes C_{+} = C_{i} and C_{−} = {C_{j}|j = 1, ⋯, K, j ≠ i}, and get the corresponding discriminant function f(x) of the binary problem:

if f_{i}(x) \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then \{ \begin{matrix}x ∈ C_{+} = C_{i} \\ x ∈ C_{−} \end{matrix}  (7.24)

which represents quantitatively how much a given x belongs to class C_{+} = C_{i}, instead of C_{−} containing the rest K −1 classes. This process is repeated K times for all i = 1, ⋯, K to get

f_{k}(x) = max{f_{1}(x), ⋯, f_{K}(x)}, (k = 1, ⋯, K)  (7.25)

which indicates x belongs to class C_{k}.

Alternatively, a multiclass problem with K > 2 can also be solved by multinomial logistic or softmax regression, which can be considered as a generalized version of the logistic regression method, based on the softmax function of K variables z_{1}, ⋯, z_{K}:

s_{k} = s(z_{k}) = \frac{e^{z_{k}}}{∑_{l=1}^{K} e^{z_{l}}} , (k = 1, ⋯, K)  (7.26)

with the following properties:

0 < s_{k} < 1,  ∑_{k=1}^{K} s_{k} = 1  (7.27)

and

\frac{∂s(z_{i})}{∂z_{j}} = \frac{δ_{ij}e^{z_{i}} ∑_{l=1}^{K} e^{z_{l}} − e^{z_{i}}e^{z_{j}}}{(∑_{l=1}^{K} e^{z_{l}})^{2}}
= δ_{ij}s(z_{i}) − s(z_{i})s(z_{j}) = s(z_{i})[δ_{ij} − s(z_{j})]  (7.28)

where δ_{ij} is the Dirac delta function which is 1 if i = j, but 0 otherwise.

Similar to the sigmoid function used to model the conditional probability p(ŷ = 1|x, w) for any data point x to belong to class C_{+} based on model parameter w in Eq. (7.6), here the softmax function s_{k} defined above is used to model the conditional probability for x to belong to class C_{k} based on some model parameter W = [w_{1}, ⋯, w_{K}]:

p(ŷ = k|x, W) = s_{k} = s(w_{k}^{T} x) = \frac{e^{w_{k}^{T} x}}{∑_{l=1}^{K} e^{w_{l}^{T} x}} , (k = 1, ⋯, K)  (7.29)

where W = [w_{1}, ⋯, w_{K}] is composed of K weight vectors each associated with one of the K classes {C_{1}, ⋯, C_{K}}, to be determined in the training process based on training set D. Same as in logistic regression, here both x and w are augmented d + 1 dimensional vectors.

We note that the inner product w^{T} x = ||w|| ||x|| cos θ of vectors w and x is inversely related to the angle θ between the two vectors, i.e., when compared with other classes, a data sample x has a smaller angular distance to w_{k}, then it has a larger inner product w_{k}^{T} x and thereby greater probability p(ŷ = k|x) to belong to class C_{k}. We therefore see that weight vectors w_{1}, ⋯, w_{K} as the parameters of the softmax regression model actually represent the angular directions of the corresponding classes in the feature space.

Figure 7.4 Softmax Classification

Specially, when K = 2, the softmax function defined in Eq. (7.29) becomes the logistic function:

s_{1} = \frac{e^{w_{1}^{T} x}}{e^{w_{0}^{T} x} + e^{w_{1}^{T} x}} = \frac{1}{1 + e^{−(w_{1}−w_{0})^{T} x}} = σ(w^{T} x)
s_{0} = \frac{e^{w_{0}^{T} x}}{e^{w_{0}^{T} x} + e^{w_{1}^{T} x}} = \frac{1}{1 + e^{(w_{1}−w_{0})^{T} x}} = σ(−w^{T} x) = 1 − σ(w^{T} x) (7.30)

where we have defined w = w_{1} − w_{0}, as the vector between the two classes of C_{0} and C_{1} (or C_{+} and C_{−}), which is the normal direction of a decision plane separating the two classes, as shown in Fig. 7.4 for a 2-D case. Any point x of unknown class can be therefore classified into either of the two classes depending on whether its projection w^{T} x onto w is positive or negative, i.e., on which side of the decision plane it is located.

For mathematical convenience, we also label each sample x_{n} in the training set by a binary vector y_{n} = [y_{n1}, ⋯, y_{nK}]^{T}, in addition to its integer labeling y_{n} ∈ {1, ⋯, K}. If y_{n} = k indicating x_{n} ∈ C_{k}, then the kth component of y_{n} is y_{nk} = 1, while other components are zero y_{nl} = 0 for all l ≠ k. Note that all components of y_{n} add up to 1. The N training samples in X = [x_{1}, ⋯, x_{N}] are also labeled by their corresponding binary labelings in Y = [y_{1}, ⋯, y_{N}].

We define the probability for x_{n} ∈ C_{k} as the softmax function based on the linear function w_{k}^{T} x_{n}:

s_{nk} = p(ŷ = k|x_{n}) = \frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} ,  (k = 1, ⋯, K, n = 1, ⋯, N) (7.31)

and the probability for x_{n} to be correctly classified into class C_{y_{n}} can be written as the following product of K factors (of which K − 1 are equal to 1):

p(ŷ = y_{n}|x_{n}, W) = ∏_{k=1}^{K} p(ŷ = k|x_{n}, W)^{y_{nk}} = ∏_{k=1}^{K} s_{nk}^{y_{nk}}  (7.32)

Our goal is to find these weight vectors in W = [w_{1}, ⋯, w_{K}] as the parameter of the softmax model for it to optimally fit the N i.i.d. data points in the training set, so that that the following likelihood is maximized:

L(W|D) ∝ p(D|W) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, W) = ∏_{n=1}^{N} (∏_{k=1}^{K} s_{nk}^{y_{nk}})

= ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} )^{y_{nk}}  (7.33)

Equivalently we can also minimize the negative log likelihood as the objective function:

J(W) = −l(W|D) = − log L(w|D) = − ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} log (\frac{e^{w_{k}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} )

= − ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} (w_{k}^{T} x_{n} − log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}w_{k}^{T} x_{n} − ∑_{k=1}^{K} y_{nk} (log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})]

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}w_{k}^{T} x_{n} − log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}]  (7.34)

The last equality is due to the fact that the last summation of K terms is independent of k and ∑_{k=1}^{K} y_{nk} = 1. Also, same as in the case of K = 2, to avoid overfitting, we could further include in this objective function extra regularization terms λ||w_{k}||^{2}/2, (k = 1, ⋯, K) to penalize large weights for smoother thresholding:

J(W) = −l(W|D) + \frac{λ}{2} ∑_{k=1}^{K} ||w_{k}||^{2}  (7.35)

To find the optimal W that minimize this objective function J(W), we find its gradient vector with respective to each of its K columns w_{i}, (i = 1, ⋯, K):

g_{i}(W) = \frac{d}{dw_{i}} J(W) = \frac{d}{dw_{i}} [−l(W|D) + \frac{λ}{2} ∑_{k=1}^{K} ||w_{k}||^{2}]

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk} \frac{d}{dw_{i}} (w_{k}^{T} x_{n}) − \frac{d}{dw_{i}} (log ∑_{l=1}^{K} e^{w_{l}^{T} x_{n}})] + λw_{i}

= − ∑_{n=1}^{N} [∑_{k=1}^{K} y_{nk}δ_{ik}x_{n} − (\frac{e^{w_{i}^{T} x_{n}}}{∑_{l=1}^{K} e^{w_{l}^{T} x_{n}}} ) x_{n}] + λw_{i}

= ∑_{n=1}^{N} (s_{in} − y_{in}) x_{n} + λw_{i} = [x_{1}, ⋯, x_{N}] [ s_{i1} − y_{i1} ; ⋮ ; s_{iN} − y_{iN} ] + λw_{i}

= [x_{1}, ⋯, x_{N}] [ r_{1} ; ⋮ ; r_{N} ] + λw_{i} = Xr_{i} + λw_{i}  (7.36)

where r_{n} = s_{in} − y_{in} is the residual, the difference between the model output s_{in} and the binary labeling y_{in}, the ith component of y_{n} that labels x_{n} as well as y_{n}.

As s_{in} is a function of all K weight vectors in W = [w_{1}, ⋯, w_{K}], so is r_{in} = s_{in} − y_{in}, and r_{i} can be explicitly expressed as r_{i}(W). We note that Eq. (7.36) takes the same form as Eq. (7.15) for K = 2. When specially K = 2 with k ∈ {0, 1}, the two equations become the same. We therefore see that logistic regression is actually a special case of softmax regression.

We stack all K such d + 1 dimensional gradient vectors g_{1}, ⋯, g_{K} together, and get a (d + 1)K dimensional gradient vector of the objective function J(W) with respect to all K parameter vectors w_{1}, ⋯, w_{K}:

g_{J} = [ g_{1} ; ⋮ ; g_{K} ]  (7.37)

based on which the optimal parameters in W that minimize the objective function J(W) can be found by the gradient descent method.

We can also find the optimal W by the Newton’s method if we can further find the Hessian matrix H_{J} as the second derivative of J(W), by taking the derivative of the ith gradient vector g_{i}(W) (i = 1, ⋯, K) with respect to the jth weight vector w_{j} (j = 1, ⋯, K) to get the second order derivative of J(W) with respect to both w_{i} and w_{j}, (i, j = 1, ⋯, K):

H_{ij} = \frac{d^{2}}{dw_{j}dw_{i}} J(W) = \frac{d}{dw_{j}} \frac{d}{dw_{i}} J(W) = \frac{d}{dw_{j}} g_{i} = \frac{d}{dw_{j}} [Xr_{i} + λw_{i}]
= X\frac{dr_{i}}{dw_{j}} + δ_{ij}λ = XJ_{ij} + δ_{ij}λ  (7.38)

where J_{ij} = dr_{i}/dw_{j} is the N × (d + 1) Jacobian matrix of r_{i} with respect to w_{j}, of which the nth row is the transpose of the following vector:

\frac{d}{dw_{j}} r_{in} = \frac{d}{dw_{j}} (s_{in} − y_{in}) = \frac{d}{dw_{j}} s_{in} = \frac{d}{dw_{j}} s(w_{i}^{T} x_{n})
= s_{in} (δ_{ij} − s_{jn}) \frac{d}{dw_{j}} (w_{j}^{T} x_{n}) = z_{n}x_{n}  (n = 1, ⋯, N) (7.39)

where we have used Eq. (7.28) and defined z_{n} = s_{in}(δ_{ij} −s_{jn}). Now the Jacobian can be written as

J_{ij} = [ dr_{i1}/dw_{j} ; ⋮ ; dr_{iN}/dw_{j} ] = [ \begin{matrix}z_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & z_{N} \end{matrix} ] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = ZX^{T}  (7.40)

where Z = diag(z_{1}, ⋯, z_{N}) is an N × N diagonal matrix. Substituting J_{r}(W) back into the expression for H_{ij}, we get:

H_{ij} = XJ_{r} + δ_{ij}λ = XZX^{T} + δ_{ij}λ = ∑_{n=1}^{N} s_{in}(δ_{ij} − s_{jn})x_{n}x_{n}^{T} + δ_{ij}λ (7.41)

Here H_{ij} is a matrix of dimension (d + 1) × (d + 1), corresponds to w_{i} and w_{j} (i, j = 1, ⋯, K). We further stack all K × K such matrices H_{ij} together to get the K(d + 1) × K(d + 1) dimensional full Hessian matrix H_{J} of J(W) with respect to all K vectors in W = [w_{1}, ⋯, w_{K}]:

H_{J} = [ \begin{matrix}H_{11} & ⋯ & H_{1K} \\ ⋮ & ... & ⋮ \\ H_{K1} & ⋯ & H_{KK} \end{matrix} ]  (7.42)

Now we can find the optimal parameter W iteratively by the Newton-Raphson method based on both H_{J} and g_{J}:

[ w_{1} ; ⋮ ; w_{K} ]_{n+1} = [ w_{1} ; ⋮ ; w_{K} ]_{n}−H_{n}^{−1} g_{n} = [ w_{1} ; ⋮ ; w_{K} ]_{n}−[ \begin{matrix}H_{11} & ⋯ & H_{1K} \\ ⋮ & ... & ⋮ \\ H_{K1} & ⋯ & H_{KK} \end{matrix} ]_{n}^{−1} [ g_{1} ; ⋮ ; g_{K} ]_{n}

(7.43)

Once W = [w_{1}, ⋯, w_{K}] is available, any unlabeled sample x of unknown class can be classified into one of the K classes with the maximum conditional probability given in Eq. (7.29):

if p(ŷ = k|x) = max_{l} p(ŷ = l|x), then x ∈ C_{k}  (7.44)

Whether we should use softmax regression or K logistic regressions for a problem of K classes C_{1}, ⋯, C_{K} depends on the nature of the classes. The method of softmax regression is suitable if the K classes are mutually exclusive and independent, as assumed by the method. Otherwise, K logistic regression binary classifiers are more suitable.

Below is a Matlab function for estimating the K weight vectors in W = [w_{1}, ⋯, w_{K}] based on the training set D = {X, y and the hyperparameter λ:

function W=softmaxRegression(X,y,lambda)
[d N]=size(X);
K=length(unique(y));    % number of classes
X=[ones(1,N); X];       % augmented data points
d=d+1;
Y=zeros(K,N);
for n=1:N
    Y(y(n),n)=1;        % generate binary labeling Y
end
W=zeros(d*K,1);         % initial guess of K weight vectors
I=eye(K);
s=zeros(K,N);
phi=zeros(K,N);         % softmax functions
gi=zeros(d,1);          % ith gradient
Hij=zeros(d,d);         % ij-th Hebbian
er=9;
tol=10^(-6);
it=0;
while er>tol
    it=it+1;
    W2=reshape(W,d,K);  % weight vectors in d x K 2-D array
    g=[];               % total gradient vector
    for i=1:K
        for n=1:N
            xn=X(:,n);  % get the nth sample
            t=0;
            for k=1:K
                wk=W2(:,k); % get the kth weight vector
                s(k,n)=exp(wk'*xn);
                t=t+s(k,n);
            end
            phi(i,n)=s(i,n)/t; % softmax function
        end
        gi=X*(phi(i,:)-Y(i,:))'+lambda*W2(:,i);   % ith gradient
        g=[g; gi];      % stack all gradients into a long vector
    end
    H=[];               % total Hessian
    z=zeros(N,1);
    for i=1:K           % for the ith block row
        Hi=[];
        for j=1:K       % for the jth block in ith row

        for n=1:N
           z(n)=phi(i,n)*((i==j)-phi(j,n));
        end
        Hij=X*diag(z)*X';
        Hi=[Hi Hij]; % append jth block
    end
    H=[H; Hi];       % append ith blow row
end
H=H+lambda*eye(d*K); % include regulation term
W=W-inv(H)*g;        % update W by Newton's method
er=norm(g);
end
W=reshape(W,d,K);        % reshape weight into d x K array
end

Here is the function for the classification of the unlabeled data samples in matrix X based on the weight vectors in W = [w_{1}, ⋯, w_{K}]:

function yhat=softmaxClassify(W,X)
[d N]=size(X);       % dataset to be classified
[d K]=size(W);       % model parameters
X=[ones(1,N); X];    % augmented data points
yhat=zeros(N,1);
for n=1:N            % for each of the N samples
xn=X(:,n);       % nth sample point
t=0;
for k=1:K
    wk=W(:,k);   % kth weight vector
    s(k)=exp(wk'*xn);
    t=t+s(k);
end
pmax=0;
for k=1:K
    p=s(k)/t;    % probability based on softmax function
    if p>pmax
        kmax=k; pmax=p;
    end
end
yhat(n)=kmax;    % predicted class labeling
end
end

Problems

1. Implement the method of logistic regression for binary classification, based on both (a) the gradient descent and (b) the Newton-Raphson methods. Then apply the code to the datasets below:

• dataset 1 /e176/homework/2ClassData.txt

• dataset 2 /e176/homework/2CocentricData.txt

• dataset 3 /e176/homework/XOR.txt Each data file contains three columns for x_{1}, x_{2}, y, respectively. Here y is the binary labeling (1 or -1), indicating to which of the two classes point x = [x_{1}, x_{2}]^{T} belongs.

Compare the performances of the two methods in terms of the number of iterations and percentage error.

Visualize your classification results by plotting all 2-D data points x_{n} in the training set and coloring them first based on their labeling y_{n} (0 or 1) to show their true class identity (e.g., red for class 1 and blue for class 2), and then based on your classification result y′_{n}. Comparing these two plots you will see how effective your classification method is.

Show your classification results in the form of a 2 × 2 confusion matrix, where the entry on the ith row and jth column (i, j = 1, 2) is the number data points labeled by y to belong to the ith class but classified to the jth class. Ideally, if all data points are classified correctly, the confusion matrix is diagonal. Also give the percentage error, defined as the ratio of the number of mis-classified data points and the total number of data points.

2. Develop Matlab code for multi-class classification based on both (a) one-vs-one method and (b) one-vs-rest method, based on logistic regression for binary classification, then carry out classification to the following 3-class datasets. Compare their results of the two methods in terms of execution time and error rates.

• dataset 1 in /e176homework/3ClassData1.txt The first three columns are for the components for each data point x_{n} = [x_{1n}, x_{2n}, x_{3n}]^{T}, and the fourth column is the corresponding labeling y_{n}, indicating to which of the K = 3 classes x_{n} belongs.

• dataset 2 in /e176/homework/3ClassData2.txt The first two columns are for the components for each data point x_{n} = [x_{1n}, x_{2n}]^{T}, and the third column is the corresponding labeling y_{n}, indicating to which of the K = 3 classes x_{n} belongs.

• The iris dataset (https://en.wikipedia.org/wiki/Iris flower data set). This dataset can be read into your code by Matlab command x=iris_dataset, where x is a 4 × 150 matrix, each of the 150 columns is for one of the 150 data samples, containing 50 samples for each of the three species of Iris (Iris setosa, Iris virginica and Iris versicolor).

Show your classification results in the form of a confusion matrix, of which the entry in the ith row and jth column is the number of data points in the training set belonging to the ith class according to the labeling y but classified into the jth class by your classifier. Also give the percentage error, defined as the number of data samples mis-classified over the total number of samples. Compare the percentage error of three methods above for each of the three datasets.

Visualize both the training dataset (the ground truth) and your classification results in 3-D space (using Matlab function scatter3), color code each data point x_{n} red, green or blue according to its ground truth class labeling y_{n}. Repeat the process but now with each data point x_{n} color coded according to the classification result y′_{n}.

3. Develop Matlab code to Implement the softmax regression method for multiclass classification, and apply it to the datasets in the previous problem. Show your results in confusion matrix and visualization. Compare your results with those obtained in the previous problem.

8 Gaussian Process Regression and Classification

8.1 Gaussian Process Regression

The Gaussian process regression (GPR) is yet another regression method that fits a regression function f(x) to the data samples in the given training set. Different from all previously considered algorithms that treat the regression model as a deterministic function f(x) that takes an explicitly specified form, the GPR treats the regression function f(x) as a stochastic Gaussian process (GP), of which the value f(x) at any point x is assumed to be a random variable with a Gaussian distribution.

We can therefore assume the joint probability density distribution (pdf) of f(X) = [f(x_{1}), …, f(x_{N})]^{T} for all i.i.d. samples in the training set D = {X, y} takes the following form:

p(f|X) = N (m_{f}, Σ_{f}) = N (0, Σ_{f})  (8.1)

where the mean vector m_{f} = 0 is assumed to be zero for convenience and without loss of generality. Based on this prior probability of the regression function f(x), we desire to find the posterior p(f_{∗}|D, X_{∗}) for the function of any unlabeled test data X_{∗} given the training set D.

We also assume the observed data y = f(x) + n is contaminated by noise n, with a zero-mean Gaussian pdf

p(n) = N (0, σ_{n}^{2}I)  (8.2)

where the diagonal covariance matrix σ_{n}^{2}I is based on the assumption that all components of n are independent of each other. We further assume noise n is independent of f, i.e., Cov[y, n] = 0. Now y = f(X) + n, as the sum of two zero-mean Gaussian variables, is also a zero mean Gaussian process.

The unknown covariance matrix Σ_{f} in Eq. (8.1) can be constructed based on the assumption that the regression function f(x) is smooth, in the sense that any two function values f(x_{m}) and f(x_{n}) are more correlated if the distance ||x_{m} − x_{n}|| between the two corresponding data points is small, but less so if the distance is large. Now the covariance of any two function values f(x_{m}) and f(x_{n}) can be modeled by the squared exponential (SE) function, also called the kernel function:

Cov[f(x_{m}), f(x_{n})] = exp (−\frac{1}{α^{2}} ||x_{m} − x_{n}||^{2}) = k(x_{m}, x_{n}) = k(x_{n}, x_{m}) (8.3)

This SE function may take any value between 1 when ||x_{m} − x_{n}|| = 0 and f(x_{m}) = f(x_{n}) are maximally correlated, and 0 when ||x_{m} − x_{n}|| = ∞ and f(x_{m}) and f(x_{n}) are least correlated.

Here α is a hyperparameter that controls the smoothness of f(x). To see this, we consider two extreme cases. First, if α → ∞, then the SE function approaches e^{0} = 1 and f(x_{m}) and f(x_{n}) are maximally correlated, i.e., f(x) is alsolutely smooth. Second, if α → 0, then the SE function approaches e^{−∞} = 0 and f(x_{m}) and f(x_{n}) are completely uncorrelated, i.e., f(x) is least smooth. We can set α to any value between 0 and 1 depending on the desired smoothness.

The smoothness of the function f(x) plays an important role as it determines whether f(x) over or underfits the sample points in the training data. If f(x) is not smooth enough, it may be affected by outlier samples due to noise and overfitting the data; on the other hand, if f(x) is too smooth, it may not reflect the valid signal variations and underfits the data. By adjusting α, a proper tradeoff between over and underfittings can be made.

Given Cov[f(x_{m}), f(x_{n})] = k(x_{m}, x_{n}) in Eq. (8.3), the covariance matrix Σ_{f} in Eq. (8.1) can be expressed as:

Σ_{f} = Cov[f, f] = [ \begin{matrix}k(x_{1}, x_{1}) & ⋯ & k(x_{1}, x_{N}) \\ ⋮ & ... & ⋮ \\ k(x_{N}, x_{1}) & ⋯ & k(x_{N}, x_{N}) \end{matrix} ] = K(X, X) = K (8.4)

where K is the kernel matrix used to model Σ_{f}. Note that the value of all diagonal components are the same k(x_{n}, x_{n}) = 1, and the farther away an off-diagonal component k(x_{m}, x_{n}) is from the diagonal (with greater ||x_{m} − x_{n}||), the lower value it takes. In general, the off-diagonal components of Σ_{f} have larger values for a smooth function f(x) than those for a less smooth function.

Now that the prior p(f|X) = N (0, Σ_{f}) is constructed based on training set X = [x_{1}, ⋯, x_{N}], we can further find the posterior of f_{∗} = f(X_{∗}) for any test dataset X_{∗} = [x_{1∗}, ⋯, x_{M∗}] containing M unlabeled samples. In the following, we will consider first the special noise-free case with n = 0 and y = f + n = f, and then the general case of non-zero noise case with f ≠ y.

In the noise-free case, as both f = y and f_{∗} are samples from the same Gaussian process, and their joint pdf conditioned on X_{∗} as well as X can be expressed as

p(f, f_{∗}|X, X_{∗}) = p ([ \begin{matrix}f \\ f_{∗} \end{matrix} ] |X, X_{∗}) = N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{f} & Σ_{ff_{∗}} \\ Σ_{f_{∗}f} & Σ_{f_{∗}} \end{matrix} ])
= N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}K & K_{∗} \\ K_{∗}^{T} & K_{∗∗} \end{matrix} ])  (8.5)

where Σ_{f} is given in Eq. (8.4), and the other three covariance matrices can be simlarly modeled by their corresponding kernel matrices:

\{ \begin{matrix}Σ_{ff_{∗}} = Cov[f, f_{∗}] = K_{∗} = K(X, X_{∗}) \\ Σ_{f_{∗}f} = Cov[f_{∗}, f] = K_{∗}^{T} = K(X_{∗}, X) \\ Σ_{f_{∗}} = Cov[f_{∗}, f_{∗}] = K_{∗∗} = K(X_{∗}, X_{∗}) \end{matrix}  (8.6)

Given this joint pdf p(f, f_{∗}|X, X_{∗}) of both f and f_{∗} in Eq. (8.5), we can further find the conditional pdf p(f_{∗}|f, X, X_{∗}) of f_{∗} given f as well as X and X_{∗} (see properties of the Gaussian distributions in Section B.1.5):

p(f_{∗}|f, X, X_{∗}) = N (m_{f_{∗}|f}, Σ_{f_{∗}|f})  (8.7)

with

\{ \begin{matrix}m_{f_{∗}|f} & = & E[f_{∗}|f] = K_{∗}^{T} K^{−1}f \\ Σ_{f_{∗}|f} & = & Cov[f_{∗}|f] = K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} \end{matrix}  (8.8)

Based on the posterior in Eq. (8.7), we can estimate f(x_{∗}) at any unlabeled sample in the test set X_{∗} in the noise-free case.

We next consider the noisy case with n ≠ 0 and y = f + n ≠ f. Now we need to find a new posterior p(f_{∗}|y, X, X_{∗}) = p(f_{∗}|X_{∗}, D) in terms of y in the training set D to replace p(f_{∗}|f, X, X_{∗}) in Eq. (8.7) in terms of f. To do so, we follow the same process above to find first the joint pdf of f_{∗} and y and then the conditional pdf of f_{∗} given y.

The joint pdf of f_{∗} and y (both zero-mean Gaussian) can be expressed as:

p([ \begin{matrix}y \\ f_{∗} \end{matrix} ] |X, X_{∗}) = N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{y} & Σ_{yf_{∗}} \\ Σ_{f_{∗}y} & Σ_{f_{∗}} \end{matrix} ])  (8.9)

where Σ_{f_{∗}} is given in Eq. (8.6) and

Σ_{y} = Cov[y, y] = Cov[f + n, f + n] = Cov[f, f] + 2 Cov[f, n] + Cov[n, n]
= Cov[f, f] + Cov[n, n] = Σ_{f} + Σ_{n} = K + σ_{n}^{2}I
Σ_{yf_{∗}} = Cov[y, f_{∗}] = Cov[f + n, f_{∗}] = Cov[f, f_{∗}] = Σ_{ff_{∗}} = K_{∗}
Σ_{f_{∗}y} = Σ_{yf_{∗}}^{T} = K_{∗}^{T}  (8.10)

where we have used the fact that Cov[f, n] = Cov[f_{∗}, n] = 0 as noise n is assumed to be independent of f. Now we have

Cov [ \begin{matrix}y \\ f_{∗} \end{matrix} ] = [ \begin{matrix}Σ_{y} & Σ_{yf_{∗}} \\ Σ_{f_{∗}y} & Σ_{f_{∗}} \end{matrix} ] = [ \begin{matrix}K + σ_{n}^{2}I & K_{∗} \\ K_{∗}^{T} & K_{∗∗} \end{matrix} ]  (8.11)

We then get the desired posterior, the conditional pdf of f_{∗} given X_{∗} as well as the training set D = {X, y}:

p(f_{∗}|X_{∗}, D) = p(f_{∗}|X_{∗}, X, y) = N (m_{f_{∗}}, Σ_{f_{∗}})  (8.12)

with the mean and covariance

\{ \begin{matrix}m_{f_{∗}} & = & E[f_{∗}|y] = K_{∗}^{T} (K + σ_{n}^{2}I)^{−1} y \\ Σ_{f_{∗}} & = & Cov[f_{∗}|y] = K_{∗∗} − K_{∗}^{T} (K + σ_{n}^{2}I)^{−1} K_{∗} \end{matrix}  (8.13)

We see that these expressions are the same as those in Eq. (5.74) previously obtained by Bayesian linear regression, i.e., the same results can be derived by two very different methods.

In the special noise-free case when n = 0 and σ_{n} = 0, the mean and covariance in Eq. (8.13) become the same as those in Eq. (8.8), and as y = f + n = f, the posterior in Eq. (8.12) becomes the same as that in Eq. (8.7):

p(f_{∗}|X_{∗}, D) = p(f_{∗}|X_{∗}, X, y) = p(f_{∗}|X_{∗}, X, f)  (8.14)

The Matlab code below is the essential part of the Gaussian process regression algorithm. It takes as the input the training set D = {X, y}, the test samples in X_{∗}, and the noise variance σ_{n}^{2}, and it generates as the output the posterior p(f_{∗}|D, X_{∗}), in terms of its mean as the GPR regression function, and its covariance matrix, of which the variances on the diagonal represent the confidence or certainty of the regression.

xstar=xmin:0.1:xmax;           % domain of function
n=length(xstar);               % number of test samples
Kss=Kernel(xstar,xstar);       % K(x*,x*), covariance of prior
[L p]=chol(Kss);               % Cholesky decomposition
for i=1:k
    plot(xstar,L'*randn(n,1))  % plot sample curves from prior
    hold on
end
hold off
x=xstar(randi(n,1,N)));           % get N random samples from xstar
y=f(xstar)+randn(1,n)*sigma_n^2;  % get their noisy labeling values
Kxx=Kernel(x,x)+sigma_n^2*eye(N); % K(x,x)
Kxs=Kernel(x,xstar);              % K(x,x*)
Ksx=Kxs';                      % K(x*,x)
fmean=Ksx*inv(Kxx)*y;          % posterior mean, regression function
fcov=Kss-Ksx*inv(Kxx)*Kxs;     % covariance of posterior
plot(xstar,fmean,xstar,fmean+diag(fcov),xstar,fmean-diag(fcov));
[L p]=chol(fcov);              % Cholesky decomposition
for i=1:k
    plot(xstar,L'*randn(n,1))  % plot sample curves drawn from posterior
    hold on
end
hold off

Here Kernel is a function that generates the squared exponential kernel matrix based on two sets of data samples in X1 and X2:

function K=Kernel(X1,X2)
a=1;            % parameter that controls smoothness of f(x)

Figure 8.1 A Noisy Sinusoid

Figure 8.2 Gaussian Process Regression with Varying Data Points

    m=length(X1);
    n=length(X2);
    K=zeros(m,n);
    for i=1:m
        for j=1:n
            d=norm(X1(:,i)-X2(:,j));
            K(i,j)=exp(-(d/a)^2);
        end
    end
end

Example 8.1 As shown in Fig. 8.1, a sinusoidal function f(x) = cos(2πfx) (with f = 0.2) contaminated by Gaussian noise with p(n) = N (0, σ_{r}^{2}) (σ_{r} = 0.2) is sampled at N = 7 random positions over the duration between the limits x_{min} = 0 and x_{max} = 9.. These samples form the training dataset D = {(x_{n}, y_{n}) (n = 1, ⋯, N)}.

The method of GPR is used to estimate the mean and variance of f_{i∗} = f(x_{i∗}) at 91 test points x_{i∗}, (i = 1, ⋯, 91), evenly distributed over the entire duration. The 8 panels of Fig. 8.2 show the estimated mean E[f_{∗}], and the variance ± Var[f_{∗}], both above and below the mean, to form a region of uncertainty of the estimates. The estimated function values are more reliable in the region that is narrow but less so if it is wide.

Figure 8.3 Samples Drawn from Posteriors

The top left panel represents the prior probability p(f_{∗}|x_{∗}) of the estimated model output f_{∗} = ŷ_{∗} at point x_{∗} with no knowledge based on the training set x (N = 0 training samples), which is assumed to have zero-mean E[f_{∗}] = 0 and unity variance Var[f(x_{∗})] = k(x_{∗}, x_{∗}) = 1. The subsequent seven panels show the mean and variance of the posterior p(f_{∗} = ŷ_{∗}|x_{∗}, D) estimated based on N = 1, ⋯, 7 training samples in D. Note that wherever an additional training sample x is available, the estimated mean μ_{f_{∗}|y} is modified to be close to the true function value f(x), and the region of uncertainty is narrowed due to the reduced variance of the posterior, i.e., the uncertainty of the estimate is reduced, or the confidence is increased. As shown in the last panel, when all N = 7 training samples are used, the general shape of the original function is reasonably reconstructed.

The 8 panels in Fig. 8.3 show 12 sample curves randomly drawn from each posterior p(f_{∗}|x_{∗}, D). They can be treated as different interpolations for the N samples in D, and any of them can be used to predict the outputs y_{∗} at any input points x_{∗}. The mean of these samples shown as the red curve is the same as E[f_{∗}] shown in the previous figure.

We see that wherever a training sample f(x) is available at x, the variance and thereby the width of the region of uncertainty is reduced, and correspondingly the sample curves in the neighborhood of x are more tightly distributed. Specifically, we see that the sample curves drawn from the prior in the top left panel are very loosely distributed, while those samples drawn from the posterior obtained based on all N = 7 training points in the bottom right panel are relatively tightly distributed. In particular, note that due to lack of training data in the interval 5 < x < 8, the region of uncertainty is significantly wider than else where, and the sample curves are much more loosely located.

The parameter α of the kernel function plays an important role in the GPR results, as shown in the three panels in Fig. 8.4, corresponding to α = 0.5, 1.5, 3.0. When α is too small (top panel), the model may overfit the data, as the regression function fits the training samples well but it not smooth and it does not interpolate regions between the training samples well (with large variance); one the other hand, when α is too large (bottom panel), the regression function is

Figure 8.4 Over Fitting (top) and Under Fitting (bottom) by GPR

smooth but it may underfit the data, as it does not fit the training samples well. By adjusting the value of α carefully we can makea a proper tradeoff between overfitting and underfitting as shown in the middle panel.

8.2 Gaussian Process Classifier – Binary

In both logistic regression and softmax regression considered previously, the linear regression function ŷ = f(x) = x^{T} w is converted to the probability p(ŷ = k|x, W) for x ∈ C_{k}, by either the logistic function σ(f(x)) for binary classification, or the softmax function s(f(x)) for multiclass classification. Also, in Gaussian process regression (GPR), the regression function f(x) is treated as a Gaussian process. Now we consider the Gaussian process classification (GPC) based on the combination of both logistic/softmax regression and GPR. We will consider binary GPC based on the logistic function in this section, and then multiclass GPC based on softmax function in the following section.

Here in binary GPC, we assume the training samples are labeled by 1 or -1 (instead of 0) for mathematical convenience, i.e., x_{n} ∈ C_{+} if y_{n} = 1 or x_{n} ∈ C_{−} if y_{n} = −1. Similar to logistic regression, here we also convert the regression function f(x), treated as a Gaussian process, into the probability for x ∈ C_{k} by the logistic function σ(f(x)), which, as a function of a random argument f(x), is also random. We therefore define p(y = 1|x, D) as the expectation of σ(f(x)) with respect to f(x):

p(y = 1|x, D) = E_{f}[σ(f(x))] = ∫ σ(f(x)) p(f|D) df  (8.15)

As function f(x) is marginalized (averaged out) by this integral, it is hidden instead of explicitly specified, and is therefore called a latent function. This p(y = 1|x, D) can be generalized for multiple samples in the training set X = [x_{1}, ⋯, x_{N}]:

p(y = 1|D) = E_{f}[σ(f(X))] = ∫ σ(f(X)) p(f|D) df  (8.16)

Here the posterior p(f|D) of f given D can be found based on Bayes’ theorem in terms of the likelihood p(y|f) = L(f|y) and the prior p(f|X):

p(f|D) = p(f|X, y) = \frac{p(y, f|X)}{p(y|X)} = \frac{p(y|f) p(f|X)}{p(y|X)} ∝ p(y|f) p(f|X)  (8.17)

As always, the denominator p(y|X) independent of f is dropped.

We first find the likelihood p(y|f). The Gaussian process f(x) is mapped by the logistic function into the probability for y = 1 representing x ∈ C_{+}, or y = −1 representing x ∈ C_{−}:

p(y = 1|f(x)) = σ(f(x))
p(y = −1|f(x)) = 1 − p(y = 1|f(x)) = 1 − σ(f(x)) = σ(−f(x)) (8.18)

These two cases can be combined to get the conditional probability of y given f(x):

p(y|f) = σ(y f) = \frac{1}{1 + e^{−yf}} = \frac{e^{yf}}{1 + e^{yf}}  (8.19)

The likelihood of f for all N i.i.d. samples in the training set X is:

p(y|f) = ∏_{n=1}^{N} p(y_{n}|f_{n}) = ∏_{n=1}^{N} σ(y_{n} f_{n})  (8.20)

We then consider the prior p(f|X), which is assumed to be a zero-mean Gaussian p(f|X) = N (0, Σ_{f}), where the covariance matrix Σ_{f} is constructed based on the training set X, the same as in GPR. Specifically, the covariance between f(x_{m}) and f(x_{n}) in the mth row and nth column of Σ_{f} is modeled by the squared exponential (SE) kernel:

Cov[f(x_{m}), f(x_{n})] = k(x_{m}, x_{n}) = exp (−\frac{1}{a^{2}} ||x_{m} − x_{n}||^{2}) , (m, n = 1, ⋯, N)

(8.21)

Such a covariance matrix is desired for GPC as well as GPR, so that f(x_{m}) and f(x_{n}) are more correlated if x_{m} and x_{n} are close together, but less so if they are farther apart. However the justification for such a property is different. In GPR, this property is desired for the smoothness of the regression function; while here in GPC, this property is also desired so that function values f(x_{m}) and f(x_{n}) for two samples x_{m} and x_{n} close to each other in the feature space are more correlated and therefore the two samples are more likely to be classified into the same class, but less so if they are far apart.

Also, as discussed in GPR, here the parameter a in the SE controls the smoothness of the function. If a → ∞, the value of the SE approaches 1, then f(x_{m}) and f(x_{n}) are highly correlated and f(x) is very smooth; but if a → 0, SE approaches 0, then f(x_{m}) and f(x_{n}) are not correlated and f(x) is no longer smooth. By adjusting a, a proper tradeoff can be made between overfitting and underfitting.

Having found both the likelihood p(y|f) and the prior p(f), we can get the posterior in Eq. (8.17):

p(f|D) ∝ p(y|f) p(f|X) = p(y|f) N (0, K)
= ∏_{n=1}^{N} σ(y_{n} f_{n}) \frac{1}{(2π)^{d/2}|K|^{1/2}} exp (−\frac{1}{2}f^{T} K^{−1}f)

∝ ∏_{n=1}^{N} σ(y_{n} f_{n}) exp (−\frac{1}{2}f^{T} K^{−1}f)  (8.22)

Note that the likelihood p(y|f) is not Gaussian, as the binary labeling y of the training set X is not continuous. Consequently, the posterior p(f|D), as a product of the Gaussian prior and non-Gaussian likelihood, is not Gaussian. However, we can carry out Laplace approximation and still approximate the posterior as a Gaussian:

p(f|D) ≈ N (m_{f|D}, Σ_{f|D})  (8.23)

of which the mean m_{f|D} and covariance Σ_{f|D} are to be obtained in the following. We further get the log posterior denoted by ψ(f):

ψ(f) = log p(f|D) = ∑_{n=1}^{N} log σ(y_{n} f_{n})− \frac{N}{2} log(2π)− \frac{1}{2} log |K|− \frac{1}{2}f^{T} K^{−1}f (8.24)

The two middle terms independent of f are dropped.

Now that the posterior p(f|D) is approximated as a Gaussian, we can find the gradient vector g_{ψ}(f) and Hessian matrix H_{ψ}(f) of the log posterior:

g_{ψ}(f) = \frac{d}{df} ψ(f) = −Σ_{f|D}^{−1}(f − m_{f|D})  (8.25)
H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ}(f) = −Σ_{f|D}^{−1}  (8.26)

On the other hand, we can also find g_{ψ}(f) and H_{ψ}(f) by directly taking the first and second order derivatives of the log posterior::

g_{ψ}(f) = \frac{d}{df} ψ(f) = \frac{d}{df} log p(y|f) − \frac{d}{df} (\frac{1}{2}f^{T} K^{−1}f) = w − K^{−1}f (8.27)

H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ}(f) = \frac{d}{df} (w − K^{−1}f) = W − K^{−1}  (8.28)

where we have defined

w = \frac{d}{df} log p(y|f),  W = \frac{d^{2}}{df^{2}} log p(y|f) = \frac{dw}{df}  (8.29)

which are respectively the gradient vector and Hessian matrix of the log function

log p(y|f) = log ∏_{n=1}^{N} σ(y_{n} f_{n}) = ∑_{n=1}^{N} log σ(y_{n} f_{n}) = ∑_{n=1}^{N} log (\frac{1}{1 + exp(−y_{n}f_{n})})

(8.30)

We first find the first and second order derivatives with respect to a single component f_{n} of f = [f_{1}, ⋯, f_{N}]^{T}:

\frac{d}{df_{n}} log p(y_{n}|f_{n}) = \frac{d}{df_{n}} log σ(y_{n}f_{n}) = \frac{d}{df_{n}} log (\frac{1}{1 + exp(−y_{n}f_{n})}) = \frac{y_{n}}{1 + exp(y_{n}f_{n})}
\frac{d^{2}}{df_{n}^{2}} log p(y_{n}|f_{n}) = \frac{d^{2}}{df_{n}^{n}} log σ(y_{n}f_{n}) = \frac{d}{df_{n}} (\frac{y_{n}}{1 + exp(y_{n}f_{n})}) = \frac{− exp(−y_{n}f_{n})}{(1 + exp(−y_{n}f_{n}))^{2}} (8.31)

where we have used the facts that y_{n} = ±1 and y_{n}^{2} = 1. We then find

w = \frac{d}{df} log p(y|f) = [ d/df_{1} ; ⋮ ; d/df_{N} ] ∑_{n=1}^{N} log p(y_{n}|f_{n})

= [ d log p(y_{1}|f_{1})/df_{1} ; ⋮ ; d log p(y_{N}|f_{N})/df_{N} ] = [ ⋮_{\frac{y_{N}}{1+e^{y_{N} f_{N}}}}^{\frac{y_{1}}{1+e^{y_{1}f_{1}}}} ]  (8.32)

and

W = \frac{d^{2}}{df^{2}} log p(y|f) = [ \begin{matrix}\frac{∂^{2}}{∂f_{1}∂f_{1}} & ⋯ & \frac{∂^{2}}{∂f_{1}∂f_{N}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}}{∂f_{N} ∂f_{1}} & ⋯ & \frac{∂^{2}}{∂f_{N} ∂f_{N}} \end{matrix} ] ∑_{n=1}^{N} log p(y_{n}|f_{n})

= diag [\frac{d^{2}}{df_{1}^{2}} log p(y_{1}|f_{1}), ⋯, \frac{d^{2}}{df_{N}^{2}} log p(y_{N}|f_{N})]
= diag [\frac{−e^{−y_{1}f_{1}}}{(1 + e^{−y_{1}f_{1}})^{2}} , ⋯, \frac{−e^{−y_{N} f_{N}}}{(1 + e^{−y_{N} f_{N}})^{2}} ]  (8.33)

Equating the two expressions for H_{ψ}(f) in Eqs. (8.26) and (8.28) we get:

H_{ψ}(f) = W − K^{−1} = −Σ_{f|D}^{−1},  i.e.,  Σ_{f|D} = (K^{−1} − W)^{−1} (8.34)

Equating the two expressions for g_{ψ}(f) in Eqs. (8.25) and (8.27) we get:

g_{ψ}(f) = −Σ_{f|D}^{−1}(f − m_{f|D}) = w − K^{−1}f
i.e.,  m_{f|D} = f + Σ_{f|D} (w − K^{−1}f) = f − H_{ψ}^{−1}g_{ψ} (8.35)

Substituting Σ_{f|D} given in Eq. (8.34) into the equation and solving it for m_{f|D}, we get

m_{f|D} = f + (K^{−1} + W)^{−1}(w − K^{−1}f)  (8.36)

We recognize this is actually the iteration of Newton’s method solving equation g_{ψ}(f) = 0, in consistence with the fact that when f = m_{f|D}, the log posterior ψ(f) = log p(f|D), assumed to be Gaussian, reaches maximum with g_{ψ}(f) = 0. However, we note that in the equation above, m_{f|D} as a value of f is expressed in terms of both w and W, which in turn are functions of f as given in Eqs. (8.32) and (8.33), i.e., m_{f|D} given above is not in a closed form, as f and parameters w and W are interdependent. We instead need to carry out an iteration during which both f and the parameters w and W are updated alternatively:

f_{n+1} = f_{n} − H_{ψ}^{−1} g_{ψ} = f_{n} + (K^{−1} − W)^{−1} (w − K^{−1}f_{n})
= f_{n} + (K^{−1} − W)^{−1} [−(K^{−1} − W)f_{n} + w − Wf_{n}]
= (K^{−1} − W)^{−1} (w − Wf_{n})  (8.37)

This iterative process converges to f = m_{f|D}, at which p(f|D) is maximized, and

g_{ψ}(f) = \frac{d}{df} ψ(f) = w − K^{−1}f = 0, i.e., f = m_{f|D} = K w  (8.38)

Now that w and W as well as f = m_{f|D} are available, we can further obtain Σ_{f|D} in Eq. (8.34), and the posterior p(f|D) ≈ N (m_{f|D}, Σ_{f|D}).

Having found p(f|D), we can proceed to carry out classification of any test set X_{∗} based on the probability (similar to Eq. (8.16)):

p(y_{∗} = 1|X_{∗}, D) = E_{f}[σ(f(X_{∗}))] = ∫ σ(f(X_{∗})) p(f|X_{∗}, D) df  (8.39)

We first approximate the posterior of f_{∗} = f(X_{∗}) in the equation as a Gaussian:

p(f|X_{∗}, D) ≈ N (m_{f_{∗}}, Σ_{f_{∗}})  (8.40)

where the mean m_{f_{∗}} and covariance Σ_{f_{∗}} can be obtained based on the fact that both f_{∗} = f(X_{∗}) and f = f(X) are the same Gaussian process, i.e., their joint probability p(f, f_{∗}) is a Gaussian. The method is therefore the same as what is discussed in the method of GPR. Specifically, we take the following steps:

• Find the mean m_{f_{∗}|f} = E(f_{∗}|f) and covariance Σ_{f_{∗}|f} = Cov[f_{∗}|f] of p(f_{∗}|X_{∗}, X, f) conditioned on the latent function f, same as in Eq. (8.8) for GPR:

\{ \begin{matrix}m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f \\ Σ_{f_{∗}|f} = K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} \end{matrix}  (8.41)

• Find m_{f_{∗}} = E_{f}(f_{∗}) as the expectation of m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f conditioned on f, i.e. find the average of m_{f_{∗}|f} over f based on p(f|D) (marginalize over f):

m_{f_{∗}} = E_{f}(m_{f_{∗}|f}) = E_{f} (K_{∗}^{T} K^{−1}f) = K_{∗}^{T} K^{−1}E_{f}(f) = K_{∗}^{T} K^{−1}m_{f}
= K_{∗}^{T} K^{−1}K w_{m_{f}} = K_{∗}^{T} w_{m_{f}}  (8.42)

where m_{f} = E_{f}(f) = Kw_{m_{f}} given in Eq. (8.38).

• Find the covariance Σ_{m_{f∗|f}} = E_{f}[(m_{f_{∗}|f} − m_{f_{∗}})^{2}]. Given the covariance Σ_{f|D} = (K^{−1} −W)^{−1} of f in Eq. (8.34), we can further find the covariance of m_{f_{∗}|f} = K_{∗}^{T} K^{−1}f as a linear combination of f (Recall if y = Ax, then Σ_{y} = AΣ_{x}A^{T}):

Σ_{m_{f∗|f}} = K_{∗}^{T} K^{−1} Σ_{f|D} K^{−1}K_{∗} = K_{∗}^{T} K^{−1}(K^{−1} − W)^{−1}K^{−1}K_{∗} (8.43)

• Find the covariance Σ_{f_{∗}} of f_{∗} as the sum of Σ_{f_{∗}|f} = E[(f_{∗} − m_{f_{∗}|f})(f_{∗} − m_{f_{∗}|f})^{T}] for the variation of f_{∗} with respect to m_{f_{∗}|f}, and Σ_{m_{f∗|f}} = E_{f}[(m_{f_{∗}|f} − m_{f_{∗}})^{2}] for the variation of m_{f_{∗}|f} with respect to m_{f_{∗}}:

Σ_{f_{∗}} = Σ_{f_{∗}|f} + Σ_{m_{f∗|f}}
= (K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗}) + (K_{∗}^{T} K^{−1}(K^{−1} − W)^{−1}K^{−1}K_{∗})
= K_{∗∗} − K_{∗}^{T} K^{−1}K_{∗} + K_{∗}^{T} K^{−1}[K − K(K − W^{−1})^{−1}K]K^{−1}K_{∗}
= K_{∗∗} − K_{∗}^{T} (K − W^{−1})^{−1}K_{∗}  (8.44)

Here we have used the identity for (A + B)^{−1} given in Section A.2.4.

Now that the posterior p(f|X_{∗}, D) = N (m_{f_{∗}}, Σ_{f_{∗}}) is approximated as a Gaussian with mean and covariance given in Eqs. (8.42) and (8.44), we can finally carry out Eq. (8.16) to find the probability for the test points in X_{∗} to belong to class C_{1}:

p(y_{∗} = 1|X_{∗}, D) = ∫ σ(f_{∗}) p(f_{∗}|X_{∗}, D) df_{∗}
= E_{f}[σ(f(X_{∗}))] = σ(E_{f}(f(X_{∗}))) = σ(m_{f_{∗}}) (8.45)

Same as in GPR, the certainty or confidence of this classification result is represented by the variances on the diagonal of the covariance Σ_{f_{∗}}.

The Matlab code for the essential parts of this algorithm is listed below. Here X and y are for the training data D = {X, y}, and Xs is an array composed of test vectors. First, the essential segment of the main program listed below takes in the training and test data, generates the covariance matrices K, K_{∗}, and K_{∗∗} represented by K, Ks, and Kss, respectively. The function Kernel is exactly the same as the one used for Gaussian process regression. This code segment then further calls a function findPosteriorMean which finds the mean and covariance of p(f|X, y) based on covariance of training data (K = Cov[X] and y, and computes the mean m_{f_{∗}|y} and covariance Σ_{f_{∗}|y} based on Eqs. (8.42) and (8.44), respectively. The sign function of m_{f_{∗}|y} indicates the classification of the test data points in X_{∗}.

    K=Kernel(X,X);               % cov(f,f), covariance of prior p(f|X)
    Ks=Kernel(X,Xs);             % cov(f_*,f)
    Kss=Kernel(Xs,Xs);           % cov(f_*,f_*)
    [Sigmaf W w]=findPosteriorMean(K,y);
                                 % get mean/covariance of p(f|D), W, w

meanfD=Ks'*w;                % mean of p(f_*|X_*,D)
SigmafD=Kss-Ks'*inv(K-inv(W))*Ks;  % covariance of p(f_*|X_*,D)
ys=sign(meanfD);             % binary classification of test data
p=1./(1+exp(-meanfD));       % p(y_*=1|X_*,D) as logistic function

The function findPosteriorMean listed below uses Newton’s method to find the mean and covariance of the posterior p(f|D) of the latent function f based on the training data, and returns them in meanf and covf, respectively, together with w and W for the gradient and Hessian of the likelihood p(y|f), to be used for computing m_{f_{∗}|D} and Σ_{f_{∗}|D}.

function [meanf covf W w]=findPosteriorMean(K,y)
% K: covariance of prior of p(f|X)
% y: labeling of training data X
% w: gradient vector of log p(y|f)
% W: Hessian matrix of log p(y|f)
% meanf: mean of p(f|X,y)
% covf: covariance of p(f|X,y)
n=length(y);        % number of training samples
f0=zeros(n,1);      % initial value of latent function
f=f0;
er=1;
while er > 10^(-9)  % Newton method to get f that maximizes p(f|X,y)
    e=exp(-y.*f);
    w=y.*e./(1+e);            % update w
    W=diag(-e./(1+e).^2);     % update W
    f=inv(inv(K)-W)*(w-W*f0); % iteration to get f from previous f0
    er=norm(f-f0);            % difference between 2 consecutive f's
    f0=f;                     % update f
end
meanf=f;                      % mean of f
covf=inv(inv(K)-W);           % coviance of f
end

Example 8.2 The GPC method is trained by the two classes shown in Fig. 8.5, represented by 100 red points and 80 blue points drawn from two Gaussian distributions N (m_{0}, Σ_{0}) and N (m_{1}, Σ_{1}), where

m_{0} = [ \begin{matrix}1 \\ 0 \end{matrix} ] , m_{1} = [ \begin{matrix}−1 \\ 0 \end{matrix} ] , Σ_{0} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , Σ_{1} = [ \begin{matrix}1 & 0.9 \\ 0.9 & 1 \end{matrix} ] (8.46)

The 3-D plot of these two normalized Gaussian distributions is also shown in the figure.

The three classification results shown in the three panels in Fig. 8.6 correspond to three different values α = 0.4, 0.2, 0.03 for the kernel function. On the left, the 2-D space is partitioned into red and blue regions corresponding to the two

Figure 8.5 Gaussian Process for Classification (Binary)

classes based on the sign function of m_{f_{∗}|y}; on the right, the 3-D distributions of σ(m_{f_{∗}|y}) represent the estimated probability for x_{∗} to belong to either class (not normalized), to be compared with the original Gaussian distributions in Fig. 8.5, from which the training samples were drawn.

We see that wherever there is evidence represented by the training samples of either class in red or blue, there are high probabilities for the neighboring points to belong to either C_{1} or C_{0} represented by the positive or negative peaks in the 3-D plots. Data points far away from any evidence will have low probability to belong to either class.

We make the following observations for three different values of the parameter α in SE:

• α = 0.4 (top row), the space is partitioned into three regions, with 20 out of 180 training points misclassified. The estimated distribution is smooth.

• α = 0.2 (middle row), the space is fragmented into several more pieces for the two classes (blue islands inside the red region and vice versa), with 5 out of the 180 training points misclassified. The estimated distribution is jagged.

• α = 0.03 (bottom row), the space is partitioned into still more pieces, with all 180 training points correctly classified. The estimated distribution is spiky.

Although the error rate is lowest when α is small, the classification result is not necessarily the best as it may well be an overfitting of the noisy data. We conclude that by adjusting parameter α, we can make proper tradeoff between error rate and overfitting.

8.3 Gaussian Process Classifier – Multi-Class

The binary GPC considered previously can be generalized to multi-class GPC based on softmax function, similar to the how binary classification based on

Figure 8.6 Classification Results with Different α Values

logistic function is generalized to multi-class classification. First of all, we define the following variables for each class C_{k}, (k = 1, ⋯, K) of the K classes {C_{1}, ⋯, C_{K}}:

• Class labeling:

Binary labeling y_{k} = [y_{1k}, ⋯, y_{Nk}]^{T} indicating whether each of the N samples in the training set X = [x_{1}, ⋯, x_{N}] belongs to C_{k}:

y_{nk} = \{ \begin{matrix}1 & if x_{n} ∈ C_{k} \\ 0 & if x_{n} ∉ C_{k} \end{matrix}  (8.47)

In other words, each training sample x_{n} ∈ C_{k} is labeled by K binary variables y_{nk} = 1 and y_{nl} = 0 for all l = 1, ⋯, K, l ≠ k (same as in softmax regression), as well as an integer labeling y_{n} = k.

• Latent functions:

Latent function f_{k} = [f_{1k}, ⋯, f_{Nk}]^{T}, of which the nth component f_{nk} = f_{k}(x_{n}) (n = 1, ⋯, N) is kth Gaussian process associated with C_{k} evaluated at the nth training sample x_{n} in the training set X = [x_{1}, ⋯, x_{N}].

• Probability modeling:

Probability p_{k} = [p_{1k}, ⋯, p_{Nk}]^{T}, of which the nth component p_{nk} (n = 1, ⋯, N) is the probability for x_{n} ∈ C_{k}, modeled by the Softmax function based on f_{nk} (same as in Eq. (7.31) in softmax regression):

p_{nk} = p(ŷ = k|x_{n}) = \frac{e^{f_{nk}}}{∑_{l=1}^{K} e^{f_{nl}}} , (k = 1, ⋯, K, n = 1, ⋯, N)

(8.48)

Note that if y_{nk} = 1 then necessarily y_{nl} = 0 for all l = 1, ⋯, K but l ≠ k, i.e., the K variables y_{n1}, ⋯, y_{nK} are not independent. given p(y_{nk} = 1), we do not need to consider p(y_{nl} = 0) for any l ≠ k.

The probability for x_{n} to be correctly classified into class C_{y_{n}} can be written as the following product of K factors (of which K − 1 are equal to 1, same as Eq. (7.32) in softmax regression):

p(ŷ = y_{n}|x_{n}, f) = ∏_{k=1}^{K} p(y_{nk} = 1|x_{n}, f)^{y_{nk}} = ∏_{k=1}^{K} p_{nk}^{y_{nk}}  (8.49)

Based on y_{k}, f_{k}, and p_{k} for all k = 1, ⋯, K, we further define the following KN dimensional vectors:

y = [ y_{1} ; ⋮ ; y_{K} ] = [ y_{11} ; ⋮ ; y_{N1} ; ⋮ ; y_{1K} ; ⋮ ; y_{NK} ] , f = [ f_{1} ; ⋮ ; f_{K} ] = [ f_{11} ; ⋮ ; f_{N1} ; ⋮ ; f_{1K} ; ⋮ ; f_{NK} ] , p = [ p_{1} ; ⋮ ; p_{K} ] = [ p_{11} ; ⋮ ; p_{N1} ; ⋮ ; p_{1K} ; ⋮ ; p_{NK} ]

(8.50)

The posterior of f given the training set D can be found based on the Bayesian theorem as (same as in Eq. (8.17) in the binary case):

p(f|D) = p(f|X, y) = \frac{p(y|f) p(f|X)}{p(y|X)} ∝ p(y|f) p(f|X)  (8.51)

We first find the likelihood based on all p_{nk} (same as Eq. (7.33) in softmax regression):

L(f|y) ∝ p(y|f) = ∏_{n=1}^{N} p(ŷ = y_{n}|x_{n}, f) = ∏_{n=1}^{N} (∏_{k=1}^{K}(p_{nk})^{y_{nk}}) = ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} )^{y_{nk}}

(8.52)

As each sample x_{n} belongs to only one of the K classes, only one of {y_{n1}, ⋯, y_{nK}} can be 1 while all others are 0, consequently, the product above contains only N probabilities raised to the power of y_{nk} = 1, each for one of the N samples belonging to a certain class, while all other probabilities raised to the power of y_{nk} = 0 become 1 and not considered.

We next assume the prior probability of the latent function f_{k} for each class C_{k} to be a zero-mean Gaussian process p(f_{k}|X) = N (0, Σ_{k}), where the covariance matrix Σ_{k} is constructed based on the squared exponential (SE) for its mn-th component:

Cov[f_{k}(x_{m}), f_{k}(x_{n})] = Cov[f_{mk}, f_{nk}] = k(x_{m}, x_{n})
= exp (−\frac{1}{a^{2}} ||x_{m} − x_{n}||^{2}) , (m, n = 1, ⋯, N)(8.53)

Then the prior of f containing all K such latent functions is also a Gaussian p(f|X) = N (0, K), where 0 is a KN-dimensional zero vector and K is a KN×KN block diagonal matrix

K = [ \begin{matrix}K_{1} & 0 & ⋯ & 0 \\ 0 & K_{2} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & K_{K} \end{matrix} ]  (8.54)

with K_{1} = ⋯ = K_{K} as defined in Eq. (8.4) on the diagonal. All off-diagonal blocks are zero as the latent functions of different classes are uncorrelated.

Having found both the likelihood p(y|f) and prior p(f|X), we can write the posterior as

p(f|D) ∝ p(f|X, y) ∝ p(y|f) p(f|X) = ∏_{n=1}^{N} ∏_{k=1}^{K} (\frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} )^{y_{nk}} N (0, K)

(8.55)

We note that the likelihood p(y|f) is not Gaussian, as y_{nk} ∈ {0, 1} is a binary labeling, instead of a continuous variable. As a product of the Gaussian prior and non-Gaussian likelihood, the posterior p(f|D) is not a Gaussian process either. However, for convenience, we still assume it is approximately Gaussian p(f|D) ≈ N (f, m_{f|D}, Σ_{f|D}), of which the mean m_{f|D} and covariance Σ_{f|D} are to be found.

Taking log of the posterior, we get

ψ(f) = log p(f|D) = log p(y|f) + log p(f|X)
= ∑_{n=1}^{N} ∑_{k=1}^{K} y_{nk} (f_{nk} − log ∑_{h=1}^{K} e^{f_{nh}}) + log N (0, K)

= y^{T} f − ∑_{n=1}^{N} log ∑_{h=1}^{K} e^{f_{nh}} − \frac{NK}{2} log(2π) − \frac{1}{2} log |K| − \frac{1}{2}f^{T} K^{−1}f(8.56)

where we have used the fact that ∑_{k=1}^{K} y_{nk} = 1. The gradient of ψ(f) is

g_{ψ} = \frac{d}{df} ψ(f) = \frac{d}{df} (y^{T} f − ∑_{n=1}^{N} log ∑_{h=1}^{K} e^{f_{nh}} − \frac{NK}{2} log(2π) − \frac{1}{2} log |K| − \frac{1}{2}f^{T} K^{−1}f)
= y − p − K^{−1}f  (8.57)

where the second term p comes from

∑_{n=1}^{N} \frac{d}{df} log ∑_{h=1}^{K} e^{f_{nh}} = ∑_{n=1}^{N} [ 0 ; ⋮ ; 0 ; \frac{e^{f_{nk}}}{∑_{h=1}^{K} e^{f_{nh}}} ; 0 ; ⋮ ; 0 ] = ∑_{n=1}^{N} [ 0 ; ⋮ ; 0 ; p_{nk} ; 0 ; ⋮ ; 0 ] = p  (8.58)

Note that p = p(f) is a function of f.

As the posterior p(f|D) ≈ N (m_{f|D}, Σ_{f|D}) is approximated as a Gaussian, which reaches maximum at its mean m_{f|D}, and so does the log prior ψ(f) = log p(f|D), and the gradient of psi(f) is zero at f = m_{f|D}:

g_{ψ}(f)|_{f=m_{f|D}} = y − p − K^{−1}f|_{f=m_{f|D}} = y − p_{m_{f}} − K^{−1}m_{f|D} = 0, (8.59)

i.e.,

m_{f|D} = K(y − p_{m_{f}})  (8.60)

where p_{m_{f}} = p(m_{f|D}) is the vector defined above evaluated at f = m_{f|D}. We further get the Hessian matrix of ψ(f):

H_{ψ}(f) = \frac{d^{2}}{df^{2}} ψ(f) = \frac{d}{df} g_{ψ} = \frac{d}{df} (y − K^{−1}f − p) = −K^{−1} − W = −Σ_{f|D}^{−1}

(8.61)

The last equality is due to the property of the Gaussian distribution given in Section B.1.5, from which we get the KN by KN covariance matrix of p(f|D):

Σ_{f|D} = −H_{ψ}^{−1}(f) = (K^{−1} + W)^{−1}  (8.62)

Here W = dp/df is a KN × KN Jacobian matrix of p, of which the ijkl-th component (i, j = 1, ⋯, N, k, l = 1, ⋯, K) is:

\frac{∂}{∂f_{jl}} p_{ik} = \frac{∂}{∂f_{j}^{l}} [\frac{e^{f_{ik}}}{∑_{h=1}^{K} e^{f_{ih}}} ] = \frac{e^{f_{ik}} (∑_{h=1}^{K} e^{f_{ih}}) δ_{kl} − e^{f_{ik}}e^{f_{jl}}}{(∑_{h=1}^{K} e^{f_{ih}})^{2}} δ_{ij}

= [\frac{e^{f_{ik}}}{∑_{h=1}^{K} e^{f_{ih}}} δ_{kl} − \frac{e^{f_{ik}}e^{f_{jl}}}{(∑_{h=1}^{K} e^{f_{ih}})^{2}} ] δ_{ij} = (p_{ik}δ_{kl} − p_{ik}p_{jl})δ_{ij},(8.63)

and W can be written also in two terms:

W = diag(p) − PP^{T}  (8.64)

where the first term diag(p) is a diagonal matrix containing all KN components of p along the diagonal (k = l and i = j), and P in the second term is a KN by N matrix composed of K N × N diagonal matrices diag(p_{k}):

P = [ diag(p_{1}) ; ⋮ ; diag(p_{K}) ] , diag(p_{k}) = [ \begin{matrix}p_{1k} & 0 & ⋯ & 0 \\ 0 & p_{2k} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & p_{Nk} \end{matrix} ] , (k = 1, ⋯, K)

(8.65)

so that

PP^{T} = [ diag(p_{1}) ; ⋮ ; diag(p_{K}) ] [diag(p_{1}), ⋯, diag(p_{K})]

= [ \begin{matrix}diag(p_{1}^{2}) & ⋯ & diag(p_{1}) diag(p_{K}) \\ ⋮ & ... & ⋮ \\ diag(p_{K}) diag(p_{1}) & ⋯ & diag(p_{K}^{2}) \end{matrix} ]  (8.66)

with

diag(p_{k}) diag(p_{l}) = [ \begin{matrix}p_{1}^{k}p_{1}^{l} & 0 & ⋯ & 0 \\ 0 & p_{1}^{k}p_{1}^{l} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 & p_{Nk}p_{Nl} \end{matrix} ]  (8.67)

Having found both the gradient g_{ψ} = y−p−K^{−1}f and Hessian H_{ψ} = −(K^{−1}+ W), we can further find the mean m_{f|D} at which ψ(f) achieves maximum by the following iteration of Newton’s method:

f_{n+1} = f_{n} − H_{ψ}^{−1}g_{ψ} = f_{n} + (K^{−1} + W)^{−1}(y − K^{−1}f_{n} − p)
= f_{n} + (K^{−1} + W)^{−1}(−(K^{−1} + W)f_{n} + Wf_{n} + y − p)
= (K^{−1} + W)^{−1}(Wf_{n} + y − p)  (8.68)

We note that during the iteration, we need to update not only f, but also p as a functions of f given in Eq. (8.58).

Now that we have got both m_{f|D} and Σ_{f|D} of f, we can further get m_{f_{∗}} and Σ_{f_{∗}} of f_{∗}:

• Get mean m_{f_{∗}}:

Similar to Eq. (8.42) in the binary case, we have the following based on f = K(y − p) in Eq. (8.60):

m_{f_{∗}} = E_{f}(m_{f_{∗}|f}) = E_{f}(K_{∗}^{T} K^{−1}f) = K_{∗}^{T} K^{−1}E_{f}(f)
= K_{∗}^{T} K^{−1} [K(y − p_{m_{f}})] = K_{∗}^{T} (y − p_{m_{f}})  (8.69)

As the K classes are uncorrelated, i.e., K is block-diagonal, the above can be separated into K equations each for one of the classes:

m_{f_{∗}^{k}} = (K_{∗}^{k})^{T} (y^{k} − p^{k}), (k = 1, ⋯, K)  (8.70)

• Get covariance Σ_{f_{∗}}:

Similar to Eq. (8.44) in the binary case, we have the following based on Σ_{f|D} = (K^{−1} + W)^{−1} in Eq. (8.62):

Σ_{m_{f∗|f}} = K_{∗}^{T} K^{−1} Σ_{f|D} K^{−1}K_{∗} = K_{∗}^{T} K^{−1}(K^{−1} + W)^{−1}K^{−1}K_{∗} (8.71)

where W = diag(p)−PP^{T} is given in Eq. (8.64) with p evaluated at m_{f|D}. Then, similar to Eq. (8.44), we get

Σ_{f_{∗}} = K_{∗∗} − K_{∗}^{T} (K + W^{−1})^{−1}K_{∗}  (8.72)

Now we can further get the probability for x_{∗} to belong to C_{k} based on the softmax function in Eq. (8.48)

p_{∗}^{k} = \frac{e^{m_{f∗}^{k}}}{∑_{l=1}^{K} e^{m_{f∗}^{l}}} , (k = 1, ⋯, K)  (8.73)

and classify x_{∗} to class C_{k} if p_{∗}^{k} = max{p_{∗}^{1}, ⋯, p_{∗}^{K}}.

Moreover, the certainty or confidence of this classification result can be found from Σ_{f_{∗}}.

The Matlab code for the essential parts of the algorithm is listed below. First, the following code segment carries out the classification of n given test samples based on the N training set D = {X, y}.

K=Kernel(X,X);     % covariance of prior of p(f|X)
Ks=Kernel(X,Xs);
Kss=Kernel(Xs,Xs);
[meanf Sigmaf p W]=findPosteriorMeanMC(K,y,C);
                   % find mean and covariance of p(f|D), and W, p
Sigmafy=Kss-Ks'*inv(K+inv(W))*Ks; % covariance of p(f|D)
p=reshape(p,N,C);
y=reshape(y,N,C);
for k=1:C
    meanfD(:,k)=Ks'*(y(:,k)-p(:,k);
                   % mean of p(f_*|X_*,D) for kth class
end
for i=1:n          % for each of n test samples
    d=sum(exp(meanfD(i,:)));
                   % denominator of softmax function
    [pmax k]=max(exp(meanfD(i,:))/d);
                   % find class with max probability
    ys(i)=k;       % label ith sample as member of kth class
    pr(i,k)=pmax;  % probility of ith sample belonging to kth class

end

The code segment above calls the following function which computes the mean and covariance of p(f|X, y) by Newton’s method:

function [meanf Sigmaf p W]=findPosteriorMeanMC(K,y,C)
% get mean and covariance of p(f|X,y) by Newton's method, given {X,y}
n=length(y);         % number of training samples
f0=zeros(n,1);       % initial value of latent function
f=zeros(n,1);
er=1;
k=0;
while er > 10^(-9)   % find f that maximizes p(f|X,y)
    k=k+1;
    [p W]=findp(f,N,C);
                     % call a function to find vector p and matrix W
    f=inv(inv(K)+W)*(W*f0+y-p);
                     % iteratively update value of f
    er=norm(f-f0);   % difference between consecutive iterations
    f0=f;            % update f
end
meanf=f;
Sigmaf=inv(inv(K)+W);
end

The following function called by the previous function finds vector p and matrix W = diag(p) − PP^{T}:

function [p W]=findp(f,N,C)  % find vector p and matrix W=diag(p)-P*P'
F=reshape(f,N,C)';       % kth row contains N samples of class k
p=zeros(C,N);            % initialize p
for n=1:N                % for each of N training samples
    d=sum(exp(F(:,n)));  % sum of all C terms in denominator
    for k=1:C            % for all C classes
        p(k,n)=exp(F(k,n))/d;
    end
end
P=[];
for k=1:C                % generate P
    P=[P; diag(p(k,:))]; % stack C diagonal matrices
end
p=reshape(p',N*C,1);     % convert p into a column vector
W=diag(p)-P*P';          % generate W matrix
end

Figure 8.7 Gaussian Process for Classification (Multi-Class)

Example 8.3 This example illustrates the classification of the same dataset of three classes used before. The top two panels of Fig. 8.7 show the distributions of the three classes in the training data set, while the bottom two panels show the classification results in terms of the partitioning of the feature space (bottom left) and the posterior distribution p(f_{∗}|X, y, X_{∗}) (bottom right), which can be compared with the distribution of the training set (top right). The confusion matrix of the classification result is shown below, with the error rate 30/600 = 0.045:

[ \begin{matrix}197 & 3 & 0 \\ 0 & 195 & 5 \\ 1 & 21 & 178 \end{matrix} ]

Example 8.4 This example illustrates the classification of the XOR data set as shown in Fig. 8.8. The confusion matrix of the classification result is shown below, with the error rate 13/400 = 0.0325:

Figure 8.8 Gaussian Process Classification for the XOR Problem

[ \begin{matrix}193 & 7 \\ 6 & 194 \end{matrix} ]

We see that in both examples, the error rates of the GPC method are lower than those of the naive Bayesian method. However, the naive Bayesian method does not have the overfitting problem, while in the method of GPC, we may need to carefully adjust the parameter of squared exponential for the kernel functions to make proper tradeoff between overfitting and error rate.

Problems

1. Implement the method of Gaussian process regression and carry out regression based on the dataset in problem as in Example 8.1 with N = 8 training samples randomly drawn from a noisy sinusoidal function y = f(x) + n = cos(2πfx) + n in the domain x=0:0.1:9. Here f = 0.2 is the frequency and the noise has a Gaussian distribution with (μ_{n} = 0, σ_{n}^{2}). Try different values of σ (e.g., σ = 0.2, 0.5, 1, ⋯).

Plot your results in terms of the mean E[f_{∗}] for the regression function and variance ± Var[f_{∗}] for the range of certainty as three curves, the same as in Figs. 8.2 and 8.3.

Try different values for the hyperparameter α (variable a in Eq. (8.3) to see how the smoothness of the regression function is affected, and to make a proper tradeoff between underfitting and overfitting.

2. Implement the binary Gaussian process classifier and then carry out binary classification of the dataset below containing data points belonging to either of two classes (labeled by 1 or -1).

/e176/programs/GPBCdata.txt

Try three different α values for the squared exponential function to show underfitting, overfitting and proper fitting (similar Example 8.2).

3. Implement the Multi-Class Gaussian process classifier and then carry out multi-class classification of the dataset below: containing data points belonging to any of three classes (labeled by 1, 2 or 3).

/e176/programs/GPMCdata.txt

Try three different α values for the squared exponential function to show underfitting, overfitting and proper fitting.

Part III Feature Extraction

In Part III, we will discuss various methods for feature selection or extraction, which can be considered as a preprocessing stage for the task of pattern classification, one of the main tasks in machine learning to be discussed in Part IV, for the goal of classifying a set of patterns into one of some K classes or clusters. Here the generic term pattern meanss any objects of interest each described by a set of d features denoted as a column vector x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional feature space, of which each component is some value that quantitatively measures the corresponding feature used to describe the pattern. The task of classification can therefore be considered as to partition the feature space into K regions each corresponding to one of the K classes or clusters.

Obviously the computational complexity of an operation on the dataset is a function (e.g., linear, polynomial, expential, etc.) of the dimensionality d of the feature space. When d is large, it is often desirable to reduce it to some significantly smaller value d′ ≪ d, while still maintaining most of the information in the dataset relevant to the task at hand, such as the variation of the features that describe the patterns or distinguish different classes of the patterns. For example, in machine learning and data science, it is often desirable to visualize the dataset in a high (d > 3) dimensional featue space to gain some intuitive sense of the dataset. It is therefore necessary to reduce the dimensionality to three or even two while maintaining the signal variation of interest in the data as much as possible. As another example in classification, the computational complexity of an algorithm may be too high if the dimensionality of the feature space is high. To reduce the computational complexity, we may consider reducing the dimensionality while still preserving the separability of the data points of different classes.

Feature selection or feature extraction can be used to reduce the dimensionality of a given dataset, by either selecting d′ features directly from the d original ones (with C_{d}^{d′} = d!/d′!(d−d′)! ways to do so), or generating d′ new features each as a function, such as the linear combinations, of the d original ones. In either case, the essential information of interest in the data needs to be maximally maintained in the resulting lower dimensional space spanned by the d′ new features.

In the following chapters we will first consider how information of interest, such as class separability, can be measured quantitatively, and then discuss specific methods for information extraction and dimensionality reduction, such as the methods of principal component analysis (PCA), independent component analysis (ICA) and their variations, widely used in signal processing, data science and machine learning.

9 Feature Selection

9.1 Distances and Separability Measurements

Before considering various feature selection methods we first define some typical distances used to quantitatively measure the difference between two entities, such as data points and groups of data points. In the following, we assume there are N samples in the dataset, of which N_{k} samples belong to class C_{k}, i.e., N_{1} + ⋯ + N_{K} = N, and we define P_{k} = N_{k}/N as the prior probability for any sample x randomly selected from the dataset to belong to C_{k}.

• Distance between two data points

The distance between two points x and y in the d-dimensional feature space can be measured by the p-norm of the difference x − y (Eq. (A.203) in Section A.4.1):

d_{p}(x, y) = ||x − y||_{p} = (∑_{i=1}^{d} |x_{i} − y_{i}|^{p})^{1/p} 1 ≤ p ≤ ∞  (9.1)

Specially, consider the following three cases:

– p = 1, city block or Manhattan distance:

d_{1}(x, y) = ||x − y||_{1} = ∑_{i=1}^{d} |x_{i} − y_{i}|  (9.2)

– p = 2, Euclidean distance:

d_{2}(x, y) = ||x − y||_{2} = \sqrt{∑_{i=1}^{d} |x_{i} − y_{i}|^{2}}  (9.3)

– p = ∞, the Chebyshev distance:

d_{∞}(x, y) = ||x − y||_{∞} = max{|x_{1} − y_{1}|, ⋯, |x_{d} − y_{d}|}  (9.4)

• Intra-class distance

An intra-class distance measures how widely (or narrowly) all samples in each class C_{k} are distributed in the space, i.e., how tightly all data points in the class are clustered . It needs to be small for good separability.

– The max diameter:

d_{max}(C_{k}) = max_{x,y∈C_{k}} d(x, y)  (9.5)

– The average diameter:

d_{average}(C_{k}) = \frac{1}{N_{k}(N_{k} − 1)} ∑_{x,y∈C_{k},x≠y} d(x, y)  (9.6)

– The average distance from each sample to the mean vector of the cluster:

\frac{1}{N_{k}} ∑_{x∈C_{k}} d(x, m_{k}), where m_{k} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x  (9.7)

– The covariance of all samples in the class represents the tightness of the samples in the class:

Σ_{k} = \frac{1}{N_{k}} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T}  (9.8)

and its determinant or trace can be used as such a scalar measurement of the tightness:

det Σ_{k} = ∏_{i=1}^{d} λ_{i}, tr Σ_{k} = ∑_{i=1}^{d} λ_{i}  (9.9)

As Σ_{k} is positive semi-definite, all of its eigenvalues λ_{i} are nonnegative and so are its determinant and trace.

• Distance between a point and a cluster/class of points

The distance between a single point x and a cluster C_{k} can be defined differently depending on how the cluster is characterized.

– The max and min-distances: This distance assumes no additional knowledge of the cluster except its member data points.

d_{max}(x, C_{k}) = max_{y∈C_{k}} d(x, y), d_{min}(x, C_{k}) = min_{y∈C_{k}} d(x, y) (9.10)

– The centroid distance: This distance assumes each cluster C_{k} of points is characterized by their mean m_{k} representing their central location in the feature space.

d_{centroid}(x, C_{k}) = d_{2}(x, m_{k}) = ||x − m_{k}||^{2} = (x − m_{k})^{T} (x − m_{k})

(9.11)

– The Mahalanobis distance: This distance assumes each cluster C_{k} of points is characterized by their covariances in Σ_{k} representing how they are distributed (the tightness-looseness, the shape and orientation of their distribution) as well as their mean m_{k} for the central position.

d_{M}(x, C_{k}) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})  (9.12)

Note that this distance is positively related to the distance between x and m_{k}, but negatively related to Σ_{k}, i.e., the more tightly they are clustered, the greater the distance d_{M}(x, C_{k}).

• Inter-class distance

An intra-class distance measures the difference between two classes, it needs to be large for good separability.

– The max and min-distances:

d_{max}(C_{i}, C_{j}) = max_{x∈C_{i},y∈C_{j}} d(x, y), d_{min}(C_{i}, C_{j}) = min_{x∈C_{i},y∈C_{j}} d(x, y)

(9.13)

– The average distance between two clusters is the average of all pair-wise distances (e.g., Euclidean) between members of the two classes:

d_{average}(C_{i}, C_{j}) = \frac{1}{N_{i}N_{j}} ∑_{x∈C_{i},y∈C_{j}} d(x, y)  (9.14)

– The centroid distance is the distance (e.g., Euclidean) between the centroids (mean vectors) of the two classes:

d_{centroid}(C_{i}, C_{j}) = d(m_{i}, m_{j})  (9.15)

– The Bhattacharyya distance:

d_{B}(C_{i}, C_{j}) = \frac{1}{4}(m_{i} − m_{j})^{T} (\frac{Σ_{i} + Σ_{j}}{2} )^{−1} (m_{i} − m_{j})
+ log [\frac{|(Σ_{i} + Σ_{j})/2|}{(|Σ_{i}| |Σ_{j}|)^{1/2}} ]  (9.16)

Similar to the the Mahalanobis distance, the first term is for the difference between the mean vectors of the two clusters but negatively related to the average of their covariances. The second term reflects the difference between the covariances of the two clusters, which is always positive due to the AM-GM inequality (algebraic mean vs geometric mean) of the arithmetic and geometric means:

\frac{1}{n} ∑_{i=1}^{n} x_{i} ≥ (∏_{i=1}^{n} x_{i})^{1/n}  (9.17)

Note that even when m_{i} = m_{j} and therefore the first term is zero, the Bhattacharyya distance is still greater than zero if Σ_{i} ≠ Σ_{j}.

The inter and intra-class distances can also defined based on a set of scatter matrices:

• Total scatter matrix:

Thee total scatter matrix is the same as the covariance matrix of the entire dataset:

S_{T} = \frac{1}{N} ∑_{all x}(x − m)(x − m)^{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m)(x − m)^{T}(9.18)

Due to the total mean ∑x x = m, rank S_{T}) ≤ N − 1. The equality holds if all N samples are independent.

• Within-class (intra-class) scatter matrix:

S_{W} = ∑_{k=1}^{K} P_{k}Σ_{k} = ∑_{k=1}^{K} \frac{N_{k}}{N} Σ_{k} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T}(9.19)

Due to the K means m_{k} = ∑_{x∈C_{k}} x/N_{k}, (k = 1, ⋯, K) as K constraints, rank(S_{W}) = ∑_{k=1}^{K}(N_{k} − 1) ≤ N − K. Again, the equality holds if all N samples are independent.

• Between-class (inter-class) scatter matrix:

S_{B} = ∑_{k=1}^{K} P_{k}(m_{k} − m)(m_{k} − m)^{T} = ∑_{k=1}^{K} \frac{N_{k}}{N} (m_{k} − m)(m_{k} − m)^{T} (9.20)

Due to total mean m = ∑_{k=1}^{K} N_{k}m_{k}/N as an additional constraint, rank(S_{B}) ≤ K − 1.

We can show that the total scatteredness of the dataset is contributed by the within-class and between-class scatteredness, i.e., S_{T} = S_{W} + S_{B}:

S_{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m)(x − m)^{T} = \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k} + m_{k} − m)(x − m_{k} + m_{k} − m)^{T}

= \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}[(x − m_{k})(x − m_{k})^{T} + (x − m_{k})(m_{k} − m)^{T}
+(m_{k} − m)(x − m_{k})^{T} + (m_{k} − m)(m_{k} − m)^{T}]
= \frac{1}{N} ∑_{k=1}^{K} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T} + \frac{1}{K} ∑_{k=1}^{K} ∑_{x∈C_{k}}(m_{k} − m)(m_{k} − m)^{T}

= ∑_{k=1}^{K} \frac{N_{k}}{N} Σ_{k} + ∑_{k=1}^{K} \frac{N_{k}}{N} (m_{k} − m)(m_{k} − m)^{T} = S_{W} + S_{B}  (9.21)

where the two middle terms are both zero as ∑_{x∈C_{k}}(x − m_{k}) = 0.

For better separability, we want the within-class scatteredness to be small but the between-class scatteredness to be large. However, as these scatter matrices cannot be directly compared in terms of their sizes, we instead consider their traces (or determinants) as the scalar measurements for the separability:

J_{B} = tr S_{B} = tr [∑_{k=1}^{C} P_{k}(m_{k} − m)(m_{k} − m)^{T} ]

= ∑_{i=1}^{C} P_{k} tr [(m_{k} − m)(m_{k} − m)^{T} ] = ∑_{i=1}^{C} P_{k} ||m_{k} − m||^{2} (9.22)

and

J_{W} = tr S_{W} = tr [∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}}(x − m_{k})(x − m_{k})^{T} ]

= ∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}} tr [(x − m_{k})(x − m_{k})^{T} ] = ∑_{k=1}^{K} P_{k} ∑_{x∈C_{k}} ||x − m_{k}||^{2}(9.23)

We see that J_{B} is the weighted average of the Euclidean distances between m_{k} and total m for the entire dataset containing all K classes, and J_{W} is the weighted average of the Euclidean distances between all x ∈ C_{k} to m_{k} for all K classes C_{1}, ⋯, C_{K}. We also recall that the trace or determinant of a matrix is the sum or product of all of its eigenvalues (Eq. (A.105) in Section A.3.1).

To achieve a high separability, we desire to maximize tr S_{B} and minimize tr S_{W}. In the same space with a constant S_{T} = S_{B} + S_{W}, maximizing tr S_{B} is equivalent to minimizing tr S_{W}. However, to measure the separability across different spaces with different S_{T}, we need to normalize S_{B} by either S_{T} or equivalently S_{W}, and redefine the objective function as either of the following:

J_{B/W} = tr(S_{W}^{−1}S_{B}) = tr(S_{B}S_{W}^{−1}),  J_{B/T} = tr(S_{T}^{−1}S_{B}) = tr(S_{B}S_{T}^{−1}) (9.24)

To reduce the dimensionality of the feature space while maximally conserving the separability in the original d-D space, we can select d′ < d of the d original features that maximize either J_{B/W} of J_{B/T} as an objective function.

Problems

In many of the problems in this homework and subsequent ones it is required to develop code to carry out feature extraction, dimensionality reduction, and classification/clustering, to be tested on both real and synthetic datasets. The sample code below can be used to generate a synthetic dataset containing multiple classes each composed of a set of normally distributed data points, based on user specified parameters including dimensionality d, number of classes K, and number of samples in each class (adding up to the total number of samples N), as well as the means and covariances (for the positions and shape of the normal distribution respectively). The code generates a d×N array X for all data points and an N-D vector y for the labelings, to be used as the ground truth for any supervised learning algorithm. Note that the repeatability of the algorithm is desired, a fixed seed for the random number generator needs to be specified.

function [X y]=sampleDataset
    K=4;                 % number of classes
    d=3;                 % dimensionality of dataset
    rng(1);              % seed for random number generator
    Nk=[150 200 250];    % number of samples in each class

N=sum(Nk);           % totall number of samples
mu=zeros(d,K);       % mean vectors for all classes
Sigma=zeros(d,d,K);  % covariance matrices for all classes
Sigma(:,:,1)=eye(3); % covariance matrix for each class
Sigma(:,:,2)=eye(3); % feel free to specify any other form
Sigma(:,:,3)=eye(3);
mu(:,1)=[0; 0; 0];   % specify mean vector for each class
mu(:,2)=[0; 0; 1];
mu(:,3)=[0; 1; 0];
mu=mu*4;             % scaling factor (for varying separability)
X=[];                % for all data samples (d by N matrix)
y=[];                % for their labelings (1 by N vector)
for k=1:K            % for each of K classes
    Xk=mvnrnd(mu(:,k),Sigma(:,:,k), Nk(k))';
                     % generate normally distributed dataset
    X=[X Xk];        % concatenate all samples to X
    y=[y k*ones(1,Nk(k))];  % concatenate their labeling to y
    switch k
        case 1
            color='r';
        case 2
            color='g';
        case 3
            color='b';
    end
    scatter3(Xk(1,:),Xk(2,:),Xk(3,:),[],color,'.') % visualize
    hold on
end
hold off
data=[X; y];
save('data34.txt','data','-ASCII') % save dataset as ASCII file
end

The code below can be used to read in the data file generated by the code above, and any dataset composed of data array X and the labeling vector y with data points in arbitrary order.

data=load('data34.txt','data');  % read disk file into data
d=size(data,1)-1       % dimensionality of data
N=size(data,2);        % total number of samples
X=data(1:d,:);         % all N d-dimensional sample vectors
y=data(d+1,:);         % and their labelings
K=length(unique(y));   % number of classes
for k=1:K              % for each of the K classes
    idx=find(y==k);    % indices of samples in class k

        Xk=X(:,idx)        % collect all samples in class k
        mean(Xk')          % mean vector of class k
        cov(Xk')           % covariance matrix of class k
    end

In addition to synthetic data, some real datasets are also used in both homework problems and examples in the current and future chapters, such as the iris flower dataset (https://en.wikipedia.org/wiki/Iris flower data set), which was first introduced in 1936 by British statistician and biologist Ronald Fisher, and has been widely used for benchmarking various classification algorithms in recent decades. This dataset contains 150 samples of iris flowers of three different species (setosa, virginica and versicolor) each containing 50 samples characterized by four features, the length and width of sepals and petals of the flower. This dataset can be read into your Metlab code by a built-in Metlab function:

    X=iris_dataset;        % read in iris data
    [d, N]=size(data);     % dimensionality and number of samples
    y=[ones(1,50) 2*ones(1,50) 3*ones(1,50)];  % data labelings

1. Generate and visualize multi-class synthetic data generated by the code above. Try different parameter values (e.g., d = 2, 3, and K = (3, 4, 5, ⋯)), and experiment different mean vectors and covariance matrices to see how the locations and shapes of distributions of the data points are effected.

2. Generate a dataset (e.g., d = 3 and K = 5) and find all K(K − 1)/2 pair-wise distances {d_{p}(m_{i}, m_{j}), i = 2, ⋯, K, j = 1, ⋯, i − 1} between the mean vectors of any two of different classes for each value of p = 1, 2, ∞. Display your results as the lower triangular matrix (all components below the main diagonal of a K × K matrix).

3. Find all pair-wise Bhattacharyya distances between any two classes, and show them as the lower triangular matrix, and compare them with those obtained in the previous problem.

4. Find the between-class, within-class and total scatter matrices and verify they do satisfy S_{B} + S_{W} = S_{T}.

5. Repeat all problems above for the iris dataset.

10 Principal Component Analysis

The Principal component analysis (PCA) is a method widely used for visualizing, processing and analyzing large datasets of high dimensionalities. The PCA is capable of significantly reducing the dimensionality of the dataset, while still maximally keeping the signal variation or information of interest in the dataset measured in certain way, so that the subsequent processing and analysis of the dataset can be carried out much more effectively and efficiently.

As always, we assume the given dataset is in the form of a d × N matrix X = [x_{1}, ⋯, x_{N}], of which each column vector x = [x_{1}, ⋯, x_{d}]^{T} is a data sample, a point in the d-dimensional feature space. In the context of pattern classification, each component x_{i} of x is some measurement of one of the d features of a pattern. Given X, the PCA carries out a special orthogonal transformation y = A^{T} x, called Karhunen–Loève transform (KLT), in such a way that the components in the resulting vector y are completely decorrelated and the energy or information contained in the data is redistributed optimally so that most of the energy is concentrated in a small number of components called principal components. Based on these two highly desirable properties, the KLT can be used for data compression in data processing and feature extraction in pattern recognition.

10.1 Covariance and Correlation

In PCA, each pattern x = [x_{1}, ⋯, x_{d}]^{T} is treated as a random vector of which each component x_{i} is a random variable with mean and variance

μ_{i} = E[x_{i}] = ∫ x_{i} p(x_{i}) dx_{i}
σ_{i}^{2} = E[(x_{i} − μ_{i})^{2}] = E[x_{i}^{2}] − μ_{i}^{2} = ∫ x_{i}^{2} p(x_{i}) dx_{i} − μ_{x_{i}}^{2}
(i = 1, ⋯, d)  (10.1)

and the covariance of x_{i} and x_{j} is:

σ_{ij}^{2} = E[(x_{i} − μ_{i})(x_{j} − μ_{j})] = E[x_{i}x_{j}] − μ_{i}μ_{j}
= ∫ ∫ x_{i}x_{j} p(x_{i}, x_{j}) dx_{i} dx_{j} − μ_{i}μ_{j}  (i, j = 1, ⋯, d) (10.2)

The mean vector and covariance matrix of x are:

m_{x} = E[x] = [ μ_{1} ; ⋮ ; μ_{d} ]

Σ_{x} = E[(x − m_{x})(x − m_{x})^{T}] = E[xx^{T}] − m_{x}m_{x}^{T}
= [ \begin{matrix}σ_{11}^{2} & ⋯ & σ_{1d}^{2} \\ ⋮ & ... & ⋮ \\ σ_{d1}^{2} & ⋯ & σ_{dd}^{2} \end{matrix} ]  (10.3)

Usually the joint probability density function p(x) = p(x_{1}, ⋯, x_{d}) of the random vector x is unknown, and the mean vector m_{x} and covariance matrix Σ_{x} of x can only be estimated by the method of maximum likelihood estimation (MLE) (Section B.3.3) based on a set of observed data samples in data set X:

m̂_{x} = \frac{1}{N} ∑_{n=1}^{N} x_{n},  Σ̂ = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} −m̂)(x_{n} −m̂)^{T} = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T} −m̂m̂^{T}

(10.4)

Note that the rank of the d × d estimated covariance matrix Σ̂ is at most N − 1, due to the N samples in the dataset X, assumed to be are independent, and the additional constraint:

∑_{n=1}^{N}(x_{n} − m̂) = ∑_{n=1}^{N} x_{n} − Nm̂ = 0  (10.5)

The variance σ_{i}^{2} = E[(x_{i} − μ_{x_{i}})^{2}] can be treated as the dynamic energy contained in x_{i}, or the amount of information carried in x_{i}, while the trace tr Σ_{x} = ∑_{i=1}^{d} σ_{i}^{2} can be considered as the total amount of dynamic energy contained in x. Also, the covariance σ_{ij}^{2} = E[(x_{i} − μ_{x_{i}})(x_{j} − μ_{x_{j}})] can be considered as the mutual energy, a measure of the correlation between x_{i} and x_{j}. By normalizing the covariance σ_{ij}^{2}, we get the correlation coefficient between x_{i} and x_{j}:

ρ_{ij} = \frac{σ_{ij}^{2}}{\sqrt{σ_{i}^{2} σ_{j}^{2}}} = \frac{σ_{ij}^{2}}{σ_{i} σ_{j}}  (10.6)

that measures how the two random variables x_{i} and x_{j} are correlated.

• ρ_{ij} = ±1 or |ρ_{ij}| = 1: x_{i} and x_{j} are maximally correlated. The information contained in the two variables is completely redundant, i.e., given the value of one of them, the value of the other is known.

• −1 < ρ_{ij} < 1 or |ρ_{ij}| < 1: x_{i} and x_{j} are correlated to different extents. The information they each carry has certain redundancy.

• ρ_{ij} = 0: x_{i} and x_{j} are uncorrelated. They each carry their own independent information with no redundancy.

Figure 10.1 Normally distributed datasets with different covariance matrices

We see that the correlation coefficient ρ_{ij} that measures how much two random variables are correlated also measures the redundancy of the information they carry. When there exists some data redundancy in the data, it is possible to carry out data compression by some method such as the PCA based on the covariance Σ to reduce the data redundancy, so that the data size can be significantly reduced while the information (dynamic energy) contained in the data is still mostly conserved.

For example, six normally distributed 2-D datasets are generated with zero mean and the following covariance matrices:

Σ_{1} = [ \begin{matrix}1.0 & 0.95 \\ 0.95 & 1.0 \end{matrix} ] , Σ_{2} = [ \begin{matrix}1.0 & −0.95 \\ −0.95 & 1.0 \end{matrix} ] , Σ_{2} = [ \begin{matrix}1.0 & 0.7 \\ 0.7 & 1.0 \end{matrix} ]

Σ_{4} = [ \begin{matrix}1.0 & −0.7 \\ −0.7 & 1.0 \end{matrix} ] , Σ_{5} = [ \begin{matrix}3 & 0 \\ 0 & 1 \end{matrix} ] , Σ_{6} = [ \begin{matrix}1 & 0 \\ 0 & 3 \end{matrix} ]

Fig. 10.1 shows the 2-D data points together with the contour line (isoline) of the corresponding normal distribution based on each of the six covariance matrices. The correlation coefficient is also shown on top of each dataset.

10.2 Karhunen-Loève Transformation

A data point in a d-dimensional space is represented by a vector x = [x_{1}, ⋯, x_{d}]^{T}, of which the components are the coordinates along the d standard orthonormal basis vectors {e_{1}, ⋯, e_{d}} that span the d-D space:

x = [ x_{1} ; x_{2} ; ⋮ ; x_{d} ] = x_{1} [ 1 0 ⋮ 0 ] + x_{2} [ 0 1 ⋮ 0 ] + ⋯ + x_{d} [ 0 ⋮ 0 1 ] = ∑_{i=1}^{d} x_{i}e_{i}

= [ e_{1} ⋯ e_{d} ] [ x_{1} ; ⋮ ; x_{d} ] = Ix  (10.7)

where I = [e_{1}, ⋯, e_{d}] is the identity matrix, and the ith component x_{i} = e_{i}^{T} x is the projection of x onto the ith standard basis vector e_{i}.

However, as shown in Fig. 10.2, the space can also be spanned by aother orthonormal basis as the column vectors in any orthogonal matrix A = [a_{1}, ⋯, a_{d}] satisfying

a_{i}^{T} a_{j} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (10.8)

and the vector can also be equivalently represented as a weighted sum of these basis vectors:

x = ∑_{i=1}^{d} x_{i}e_{i} = ∑_{i=1}^{d} y_{i}a_{i} = [ a_{1} ⋯ a_{d} ] [ y_{1} ; ⋮ ; y_{d} ] = Ay  (10.9)

where y = [y_{1}, ⋯, y_{d}]^{T} is a vector containing all d coordinates in the directions of the basis vector {a_{i}, ⋯, a_{d}}.

The orthogonal matrix A satisfies A^{T} = A^{−1}, i.e., AA^{T} = A^{T} A = I. Premultiplying A^{T} on both sides of the equation above, we get Ax = A^{T} Ay = A^{−1}Ay = y, i.e.,

y = [ y_{1} ; ⋮ ; y_{d} ] = A^{T} x = [  a_{1}^{T} ; ⋮ ; a_{d}^{T}  ] x = [ a_{1}^{T} x ; ⋮ ; a_{d}^{T} x ]  (10.10)

where the ith coordinate y_{i} = a_{i}^{T} x is the projection of x onto the ith basis vector a_{i}. The two equations above form an orthogonal transform pair:

\{ \begin{matrix}y = A^{T} x & forward transform \\ x = Ay & inverse transform \end{matrix}  (10.11)

The basis vectors in A = [a_{1}, ⋯, a_{d}] can be considered as a rotated version

Figure 10.2 Orthogonal transform as a rotation of the coordinate system

of the standard basis in I = [e_{1}, ⋯, e_{d}], and the norm or length of the vector before and after the transform remain the same:

||x||^{2} = x^{T} x = (∑_{i=1}^{d} y_{i}a_{i})^{T} (∑_{j=1}^{d} y_{j}a_{j}) = ∑_{i=1}^{d} ∑_{j=1}^{d} y_{i}y_{j} a_{i}^{T} a_{j} = ∑_{i=1}^{d} y_{i}^{2} = ||y||^{2}

(10.12)

If x is treated as a random vector, then its linear transform y = A^{T} x is also a random vector, with the following mean and covariance:

m_{y} = E[y] = E[A^{T} x] = A^{T} E[x] = A^{T} m_{x}
Σ_{y} = E[yy^{T}] − m_{y}m_{y}^{T} = E[A^{T} xx^{T} A] − A^{T} m_{x}m_{x}^{T} A
= A^{T} [E[xx^{T}] − m_{x}m^{T}]A = A^{T} Σ_{x}A  (10.13)

The Karhunen-Loève Transform (KLT) is just a special case of the orthogonal transforms considered above, in the same form of y = V^{T} x, where the orthogonal transform matrix V = [v_{1}, ⋯, v_{d}] is the eigenvector matrix of the covariance matrix Σ_{x} of x, composed of the d normalized eigenvectors of Σ_{x}. As in general the covariance matrix Σ_{x} is symmetric and positive definite, its eigenvalues {λ_{1}, ⋯, λ_{d}} are real and positive, and its eigenvectors are orthogonal

λ_{d} ≥ ⋯ ≥ λ_{1} ≥ 0, v_{i}^{T} v_{j} = δ_{ij} = \{ \begin{matrix}0 & i ≥ j \\ 1 & i = j \end{matrix} (i, j = 1, ⋯, d)

(10.14)

i.e., V is an orthogonal matrix satisfying V^{T} = V^{−1} or V^{T} V = I.

The d eigenvalues {λ_{1}, ⋯, λ_{d}} and the corresponding eigenvectors {v_{1}, ⋯, v_{d}} can then be found by solving the eigenequations:

Σ_{x}v_{i} = λ_{i}v_{i} (i = 1, ⋯, d)  (10.15)

Note that the eigenvector v_{i} is not unique, as the eigenequation still holds when v_{i} is arbitrarily scaled by any constant. The d eigenequations above can be combined to be expressed in matrix form:

Σ_{x}V = Σ_{x}[v_{1}, ⋯, v_{d}] = [v_{1}, ⋯, v_{d}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{d} \end{matrix} ] = VΛ  (10.16)

where Λ = diag(λ_{1}, ⋯, λ_{d}) is a diagonal matrix.

Premultiplying v_{i}^{T} on both sides of the eigenequation in Eq. (10.15), we get:

v_{i}^{T} Σ_{x}v_{i} = λ_{i}v_{i}^{T} v_{i} = λ_{i} (i = 1, ⋯, d)  (10.17)

or in matrix form, premultiplying V^{T} = V^{−1} on both sides of Eq. (10.16), we get

V^{T} Σ_{x}V = V^{−1}VΛ = Λ  (10.18)

Taking inverse on both sides, we also get

(V^{T} Σ_{x}V)^{−1} = V^{−1}Σ_{x}^{−1}(V^{T})^{−1} = V^{T} Σ_{x}^{−1}V = Λ^{−1}  (10.19)

We see that Σ_{x}^{−1} and Σ_{x} share the same eigenvectors, their eigenvalues are reciprocal of each other, and they are both diagonalized by the orthogonal eigenvector matrix V to become Λ and Λ^{−1} respectively.

Based on the orthogonal eigenvector matrix V, the KLT is defined as:

y = [ y_{1} ; ⋮ ; y_{d} ] = V^{T} x = [  v_{1}^{T} ; ⋮ ; v_{d}^{T}  ] x = [ v_{1}^{T} x ; ⋮ ; v_{d}^{T} x ]  (10.20)

of which the ith component y_{i} = v_{i}^{T} x is the projection of vector x onto the ith eigenvector v_{i}. Premultiplying V = (V^{T})^{−1} on both sides, we get

Vy = VV^{T} x = VV^{−1}x = x  (10.21)

The two equations above form the KLT transform pair:

\{ \begin{matrix}y = V^{T} x & forward transform \\ x = Vy & inverse transform \end{matrix}  (10.22)

10.3 Optimality of KLT

The KLT is the optimal orthogonal transform among all possible orthogonal transforms y = A^{T} x based on any orthogonal matrix A satisfying A^{T} A = I, in the sense that the KLT completely decorrelates the signal, and it maximally compacts the energy (information) contained in the signal, as shown below.

• KLT Completely Decorrelates Signal Components

The mean vector and covariance matrix of the random vector y = V^{T} x after the KLT transform can found as

m_{y} = E[y] = E[V^{T} x] = V^{T} E[x] = V^{T} m_{x}  (10.23)
Σ_{y} = E[(y − m_{y})(y − m_{y})^{T}] = E[V^{T} (x − m_{x})][V^{T} (x − m_{x})]^{T}
= V^{T} E [(x − m_{x})(x − m_{x})^{T} ] V = V^{T} Σ_{x}V = Λ
= [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{d} \end{matrix} ] = [ \begin{matrix}σ_{y_{1}}^{2} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & σ_{y_{d}}^{2} \end{matrix} ]  (10.24)

Based on the fact that the covariance matrix Σ_{y} is a diagonal matrix, we have the following two observations:

– The zero off-diagonal component σ_{ij}^{2} = 0 (i ≠ j) for the covariance between any two components y_{i} and y_{j} indicates they are completely decorrelated after the KLT, i.e., each component carries its own independent information;

– The nonzero diagonal component for the variance λ_{i} = σ_{i}^{2} = E[(y_{i} − μ_{y_{i}})^{2}] (i = 1, ⋯, c) of the ith component y_{i} represents the dynamic energy or amount of information contained in y_{i}. As Σ_{x} and Σ_{y} = Λ share the same eigenvalues {λ_{1}, ⋯, λ_{d}}, and therefore have the same trace

E_{x} = tr Σ_{x} = ∑_{i=1}^{d} λ_{i} = tr Σ_{y} = E_{y}  (10.25)

We see that the total amount of energy or information contained in the signal x before the KLT and y after the KLT remains the same, i.e., the energy or information contained in the signal is conserved by the KLT. This is generally true for all orthogonal transforms y = A^{T} x (such as the Fourier transform), due to the fact that the eigenvalues remain the same under any orthogonal transform B = R^{T} AR, i.e., matrices A and B share the same eigenvalues, and thereby the same trace and determinant.

• KLT Optimally Compacts Signal Energy

While the total energy contained in the signal is conserved by any orthogonal transform, the distribution of this energy among the d components before and after the transform may be very different. We can show that the KLT is optimal in the sense that it redistributes the total energy in such a way that it is maximally compacted into a subset of components of y = V^{T} x.

Consider a generic orthogonal transform y = A^{T} x based on an arbitrary orthogonal matrix A = [a_{1}, ⋯, a_{d}] satisfying A^{T} = A^{−1}:

y = [ y_{1} ; ⋮ ; y_{d} ] = A^{T} x = [ a_{1}^{T} ; ⋮ ; a_{d}^{T}  ] x  (10.26)

We also have

y_{i} = a_{i}^{T} x,  μ_{y_{i}} = E(y_{i}) = a_{i}^{T} E(x) = a_{i}^{T} m_{x},  (i = 1, ⋯, d)

(10.27)

We then consider the energy contained in d′ < d out of the d components of y = A^{T} x expressed as a function of the transform matrix A = [a_{1}, ⋯, a_{d}]:

E′(A) = ∑_{i=1}^{d′} σ_{y_{i}}^{2} = ∑_{i=1}^{d′} E [(y_{i} − μ_{y_{i}})^{2}] = ∑_{i=1}^{d′} E [a_{i}^{T} (x − m_{x_{i}}) (x − m_{x_{i}})^{T} a_{i}]

= ∑_{i=1}^{d′} a_{i}^{T} E [(x − m_{x_{i}})(x − m_{x_{i}})^{T} ] a_{i}^{T} = ∑_{i=1}^{d′} a_{i}^{T} Σ_{x}a_{i}  (10.28)

We can find the optimal orthogonal transform matrix A that maximizes this partial energy E′ by solving the following optimization problem with the constraint that the column vectors of A are normalized with a_{j}^{T} a_{j} = 1:

\{ \begin{matrix}maximize: E′(A) = E′(a_{1}, ⋯, a_{d′}) \\ subject to: a_{j}^{T} a_{j} = 1 (j = 1, ⋯, d′) \end{matrix}  (10.29)

The Lagrangian function of this constrained optimization problem is

L(A) = E′(A) − ∑_{j=1}^{d′} λ_{j}(a_{j}^{T} a_{j} − 1)  (10.30)

To find the optimal A, we set partial derivative with respect to each a_{i} (i = 1, ⋯, d′) to zero:

\frac{∂}{∂a_{i}} L(A) = \frac{∂}{∂a_{i}} [∑_{j=1}^{d′}(a_{j}^{T} Σ_{x}a_{j} − λ_{j}a_{j}^{T} a_{j} + λ_{j})]

= \frac{∂}{∂a_{i}} (a_{i}^{T} Σ_{x}a_{i} − λ_{i}a_{i}^{T} a_{i}) = 2Σ_{x}a_{i} − 2λ_{i}a_{i} = 0 (10.31)

The resulting equation Σ_{x}a_{i} = λ_{i}a_{i} happens to be the eigenequation of the covariance matrix Σ_{x}, and the column vectors of A must be the eigenvectors of Σ_{x}, i.e., the optimal transform is the KLT A = V.

E′(V) = ∑_{i=1}^{d′} v_{i}^{T} Σ_{x}v_{i} = ∑_{i=1}^{d′} λ_{i} ≤ ∑_{i=1}^{d} λ_{i} = E_{y} = E_{x}  (10.32)

To maximize the partial energy E′(V) contained in the d′ components we carry out the KLT based on the eigenvectors v_{i} corresponding to the d′ largest eigenvalues of Σ_{x}: λ_{1} ≥ λ_{2} ≥ ⋯ λ_{d′} ⋯ ≥ λ_{d}.

10.4 Geometric Interpretation of the KLT

The optimality of the KLT discussed above can be demonstrated geometrically, based on the assumption that the random vector x = [x_{1}, ⋯, x_{d}]^{T} has a normal probability density function:

p(x) = N (m_{x}, Σ_{x}) = \frac{1}{(2π)^{d/2} |Σ_{x}|^{1/2}} exp [−\frac{1}{2}(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x})]

(10.33)

with mean vector m_{x} and covariance matrix Σ_{x}. The shape of this normal distribution in the d-dimensional space can be represented by an iso-hypersurface in the space determined by

N (m_{x}, Σ_{x}) = c_{0}  (10.34)

where c_{0} is some constant, or equivalently

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = c_{1}  (10.35)

where c_{1} is another constant related to c_{0}. As Σ^{−1} is positive definite as well as Σ, this equation represents an hyper ellipsoid in the d-dimensional space. In particular, when d = 2, x = [x_{1}, x_{2}]^{T}, with positive definite Σ_{x}^{−1}:

Σ_{x}^{−1} = [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ] and |Σ_{x}^{−1}| = AC − B^{2}/4 > 0  (10.36)

then the quadratic equation above becomes

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = [x_{1} − μ_{x_{1}}, x_{2} − μ_{x_{2}}] [ \begin{matrix}A & B/2 \\ B/2 & C \end{matrix} ] [ \begin{matrix}x_{1} − μ_{x_{1}} \\ x_{2} − μ_{x_{2}} \end{matrix} ]

= A(x_{1} − μ_{x_{1}})^{2} + B(x_{1} − μ_{x_{1}})(x_{2} − μ_{x_{2}}) + C(x_{2} − μ_{x_{2}})^{2} = c_{1}

representing an ellipse (instead of other quadratic curves such as hyperbola and parabola) centered at m_{x} = [μ_{1}, μ_{2}]^{T}. When d = 3, the quadratic equation represents an ellipsoid. In general when d > 3, the equation N (x, m_{x}, Σ_{x}) = c_{0} represents a hyper-ellipsoid in the d-dimensional space.

Substituting x = Vy and m_{x} = Vm_{y} into the iso-surface equation Eq. (10.35), we get the equation for the hyper-ellipsoid after the KLT:

(x − m_{x})^{T} Σ_{x}^{−1}(x − m_{x}) = [V(y − m_{y})]^{T} Σ_{x}^{−1}Vy − m_{y})
= (y − m_{y})^{T} V^{T} Σ_{x}^{−1}V(y − m_{y}) = (y − m_{y})^{T} Λ^{−1}(y − m_{y})
= ∑_{i=1}^{d} \frac{(y_{i} − μ_{y_{i}})^{2}}{λ_{i}} = ∑_{i=1}^{d} \frac{(y_{i} − μ_{y_{i}})^{2}}{σ_{y_{i}}^{2}} = c_{1}  (10.37)

Figure 10.3 Standardization of ellipsoids by the KLT

This a standardized hyper-ellipsoid. We therefore see that the KLT transform y = V^{T} x is actually a rotation of the coordinate system of the d-dimensional space, which is spanned by the standard basis {e_{1}, ⋯, e_{d}} before the KLT and the eigenvectors {v_{1}, ⋯, v_{d}} as the basis vectors after the KLT.

As the result, the principal semi-axes of the ellipsoid representing the Gaussian distribution of the dataset become in parallel with the axes of the new coordinate system, i.e., the ellipsoid becomes standardized. Moreover, the length of the ith principal semi-axis is proportional to the standard deviation σ_{y_{i}} = \sqrt{λ_{i}} of the ith variable y_{i}. This is the reason why KLT possesses the two desirable properties: (a) the decorrelation of the signal components, and (b) redistribution and compaction of the energy or information contained in the signal, as illustrated in Fig. 10.3.

Example 10.1 The top row of Fig. 10.4 shows three given datasets with the following parameters:

Σ_{1} = [ \begin{matrix}2.964 & 0.544 \\ 0.544 & 4.687 \end{matrix} ] , Σ_{2} = [ \begin{matrix}5.036 & 3.083 \\ 3.083 & 2.830 \end{matrix} ] , Σ_{3} = [ \begin{matrix}3.043 & −3.076 \\ −3.076 & 3.930 \end{matrix} ]

(10.38)

ρ_{1} = 0.146,  ρ_{2} = 0.817,  ρ_{3} = −0.889  (10.39)

Then the KLT is carried out to rotate the coordinate system from the orig-

Figure 10.4 Three datasets before and after the KLT

inal standard basis to the eigenvectors of their covariance matrices, with the corresponding eigenvalues shown below:

λ_{1,2} = (4.845, 2.807), λ_{1,2} = (7.207, 0.659), λ_{1,2} = (6.594, 0.379)

(10.40)

The datasets after the KLT are shown in the bottom row of the figure. We see that in each of the three cases, the ellipse representing the distribution of the data points becomes standardized, with its semi-axes proportional to the eigenvalues, which, when squared, are the variances of the principal components.

10.5 Computation of the KLT

Due to KLT’s properties of signal decorrelation and energy compaction, it can be used for data compression by reducing the dimensionality of the dataset. Specifically, we carry out the following steps:

1. Find the mean vector m_{x} and the covariance matrix Σ_{x} of the random vectors x based on the given dataset X = [x_{1}, ⋯, x_{N}] containing N samples of x. Note that the rank of Σ is no greater than N − 1.

2. Find the eigenvalues λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{d} of Σ_{x}, sorted in descending order, and their corresponding eigenvectors {v_{1}, v_{2}, ⋯, v_{d}}.

3. Determine the reduced dimensionality d′ < d so that the percentage of energy contained after the transform is no less than a given threshold (e.g., 95%):

\frac{∑_{i=1}^{d′} λ_{i}}{∑_{i=1}^{d} λ_{i}} = \frac{∑_{i=1}^{d′} σ_{y_{i}}^{2}}{∑_{i=1}^{d} σ_{y_{i}}^{2}} ≥ 0.95  (10.41)

4. Construct a d × d′ transform matrix composed of the d eigenvectors corresponding to the d largest eigenvalues of Σ_{x}:

V′ = [v_{1}, ⋯, v_{d′}]_{d×d′}  (10.42)

and carry out KLT based on V′:

y = [ y_{1} ; ⋮ ; y_{d′} ]_{d′×1} = V^{′T} x = [  v_{1}^{T} ; ⋮ ; v_{d′}^{T}  ]_{d′×d} [ x_{1} ; ⋮ ; ⋮ ; x_{d} ]_{d×1}

(10.43)

This is a lossy compression by which the dimensionality is reduced from d to d′ < d, with the following error representing the percentage of information lost:

∑_{i=d′+1}^{d} λ_{i}/ ∑_{i=1}^{d} λ_{i}  (10.44)

But as λ_{d′+1}, ⋯, λ_{d} are the smallest d − d′ eigenvalues, the error is minimum (e.g., 5%).

5. Reconstruct x by inverse KLT:

x = [ x_{1} ; ⋮ ; ⋮ ; x_{d} ]_{d×1} = V′y = [ v_{1} ⋯ v_{d′} ]_{d×d′} [ y_{1} ; ⋮ ; y_{d′} ]_{d′×1} = ∑_{i=1}^{d′} y_{i}v_{i}

(10.45)

The rank of the d × d estimated covariance matrix Σ̂_{x} in Eq. (10.4) based on the dataset X = [x_{1}, ⋯, x_{N}] is no greater than N − 1. When N ≤ d, then the rank of Σ̂ is at most N − 1 < d, i.e., there are no more than N − 1 non-zero eigenvalues while all remaining d − (N − 1) = d − N + 1 eigenvalues are zero. In this case, this estimated covariance matrix Σ̂_{x} is not invertible.

Further more, if the dimensionality d ≫ N is much higher than the number of data samples, the computational complexity O(d^{3}) for solving this eigenvalue problem of Σ̂_{x} may be expensive. In this case, we can find the N − 1 non-zero eigenvalues and their corresponding eigenvectors of Σ̂_{x} by solving the eigenvalue problem of an N × N matrix with much reduced complexity of O(N^{3}).

Specifically, for convenience and without loss of generality, we assume that the data have zero mean m_{x} = 0, so that the covariance matrix of the given dataset X can be estimate as

Σ̂_{x} = \frac{1}{N} ∑_{n=1}^{N}(x_{n} − m_{x})(x_{n} − m_{x})^{T} = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T}

= \frac{1}{N} [x_{1}, ⋯, x_{N}] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = \frac{1}{N} XX^{T}  (10.46)

and the eigenequation of this d × d matrix can be written as

Σ̂_{x}v_{i} = \frac{1}{N} XX^{T} v_{i} = λ_{i}v_{i}  (10.47)

Pre-multiplying X^{T} on both sides, we get the eigenequation of an N × N matrix X^{T} X/N:

\frac{1}{N} X^{T} X (X^{T} v_{i}) = λ_{i} (X^{T} v_{i}) i.e. \frac{1}{N} X^{T} Xu_{i} = λ_{i}u_{i}  (10.48)

where u_{i} = X^{T} v_{i} is ith eigenvector of X^{T} X/N. We see that XX^{T} /N and X^{T} X/N share the same non-zero eigenvalues. When N < d, we can solve the eigenvalue problem in Eq. (10.48) with complexity O(N^{3}), instead that of the original problem in Eq. (10.47) with O(d^{3}), to get the N −1 non-zero eigenvalues λ_{i}, (i = 1, ⋯, N − 1), and the corresponding eigenvectors u_{i} = X^{T} v_{i}. To find the eigenvectors v_{i} of the original matrix Σ̂_{x} = XX^{T} /N, we pre-multiply X on both sides of the eigenequation in Eq. (10.48) and get

\frac{1}{N} XX^{T} (Xu_{i}) = λ_{i} (Xu_{i})  (10.49)

Comparing this with Eq. (10.47), we see that Xu_{i} = v_{i}.

and they become the same when normalized.

10.6 Comparison with Other Orthogonal Transforms

Here we compare the KLT, in terms of signal decorrelation and energy compaction, with other orthogonal transforms such as discrete cosine transform (DCT) and Fourier transform (DFT), as well as identity transform IT (no transform), based on two images of cloud and sand as shown below.

We treat each column of the image as an observation (instantiation) of a 1-D random vector x, and apply an orthogonal transforms y = A^{T} x to x based on the covariance matrix Σ_{x}, and compare the corresponding covariance matrices Σ_{y} after the transform to see how well each transform decorrelates the signal and compacts its energy.

Fig. 10.5 shows two images (left) and the corresponding covariance matrices in image form (pixel values proportional to the values in the covariance matrix) after three orthogonal transforms of IT (no transform), DCT, and KLT.

Figure 10.5 Images of Clouds and Sands

• Sand image

In the second panel showing the covariance after IT, the off-diagonal pixels are dark, indicating the pixels in the original image are not highly correlated at all. In the third panel, the covariance after DCT looks similar to that before the transform, indicating that the DCT has little effect in terms of signal decorrelation and energy compaction. Finally, in the rightmost panel showing the covariance after KLT, all off diagonal elements are zero, i.e., the signal components are completely decorrelated.

• Cloud image

In the second panel for the covariance after IT, there exist some bright areas off the diagonal, indicating that many signal components close to each other are highly correlated. In the third panel for the covariance after DCT, the values of the off diagonal elements are significantly reduced, indicating that the signal components are significantly decorrelated. Finally, in the right-most panel showing the covariance after KLT, all off diagonal elements are zero, i.e., the signal components are completely decorrelated.

The effect of energy redistribution and compaction of these different transforms are also shown in Fig. 10.6, where the variances of signal components are sorted and plotted.

We see that due to the physical nature of the clouds and sand grains, the textures in the corresponding images are very different. In the cloud image, the neighboring pixels are highly correlated, while in the sand image, they are not much correlated at all. Correspondingly, the DCT can effectively decorrelate the cloud image, but much less so in the sand image. But in either case, whether the pixels in the image are highly correlated or not, the KLT can always completely decorrelate the signal and optimally compact the energy.

The table below further demonstrate the effect of energy compaction quantita-

Figure 10.6 Energy compaction and redistribution by the KLT

tively in terms of the number of components out of the total of 256 components needed after the transform in order to keep a certain percentage of the total dynamic energy (information). For example, if it is desired to keep 95% of the total energy contained in the original signal, 230 components are needed with no transform, 22 are needed after DCT, and only 13 are needed after KLT. Note that DCT’s performance is reasonably close to that of the optimal KLT in this case.

Percentage: 90% 95% 99% 100%
IT:  209 230 250 256
DCT:  10 22 97 256
KLT:  7 13 55 256

(10.50)

Although KLT is optimal in terms of signal decorrelation and energy compaction, it is not as convenient as other transforms for two reasons. First, the KLT is data-specific, i.e., the transform matrix is the eigenvector matrix of the covariance matrix of the dataset, which needs to be estimated based on sufficient amount of data samples, while all other orthogonal transforms are data independent. Second, the computational cost of the KLT is significantly higher than that of other transforms, due to the O(d^{3}) complexity for solving the eigenvalue problem of the d × d matrix covariance matrix Σ_{x}, while for most other popular orthogonal transforms fast algorithm exist with O(d log_{2} d) complexity instead of O(d^{2}) required by the KLT.

10.7 Application to Image Data

The KLT can be applied to a stack of N images each containing K pixels, as shown on the left of Fig. 10.7. Depending on the specific need (such as data compression and feature extraction), the KLT can be carried out in either of two alternative manners, which define the random signal vector x differently, as shown below.

Figure 10.7 The KLT of a set of images

• A K-dimensional vector x_{n} (n = 1, ⋯, N) can be formed by all K pixels of each of the N images by concatenating the rows (or columns) of the image array (same as what we did for the sand and cloud images). All N such vectors are treated as the column vectors of a K × N data array X = [x_{1}, ⋯, x_{N}]. The K × K covariance matrix of these K-D column vectors can be estimated (assuming with zero mean) as:

Σ = \frac{1}{N} ∑_{n=1}^{N} x_{n}x_{n}^{T} = \frac{1}{N} [x_{1}, ⋯, x_{N}] [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] = \frac{1}{N} (XX^{T} )_{K×K} (10.51)

• An N-dimensional vector can be formed by the pixels at the same position (e.g., in the ith row and jth column of the image array) from all N images. There are K such vectors each for one pixel in the images, and they are actually the rows of matrix X defined above, or the columns of X^{T}. The N × N covariance matrix of these N-D row vectors can be estimated as:

Σ′ = \frac{1}{K} (X^{T} X)_{N×N} .  (10.52)

As discussed previously, these two different covariance matrices share the same eigenvalues. The eigenequations for X^{T} X and X^{T} X (with the constant coefficients 1/K and 1/N neglected) are:

X^{T} Xv = λv,  XX^{T} u = μu.  (10.53)

Pre-multiplying X^{T} on both sides of the second equation we get

X^{T} X[X^{T} u] = μ[X^{T} u].  (10.54)

which is actually the first eigenequation with the same eigenvalue μ = λ and eigenvector v = X^{T} u when normalized. The two covariance matrices Σ and Σ′ have the same rank R = min(N, K) (if X is not degenerate) and therefore the same number of non-zero eigenvalues. Consequently, the KLT can be carried out based on either matrix with the same effects in terms of the signal decorrelation and energy compaction. As the number of pixels in the image is typically much greater than the number of images, K > N, we will take the second approach above to treat the pixels in the same position in all N images as a sample of an N-dimensional random signal vector and carry out the KLT based on the N ×N covariance matrix Σ̂.

We can now carry out the KLT to each of the K N-dimensional vectors x corresponding to each pixel of the N images to obtain another N-dimensional vector y = v^{∗}x for the same pixel of a set of N eigenimages, as shown on the right of Fig. 10.7. After the KLT, most of the energy/information contained in the N images, representing the signal variations among all N images, is concentrated in the first few eigenimages corresponding to the greatest eigenvalues, while the remaining eigenimages can be omitted without losing much energy/information. This is the foundation for various KLT-based image compression and feature extraction algorithms. The subsequent operations such as image recognition and classification can all be carried out in a much lower dimensional space.

We now consider some of such applications.

• Video image compression

A sequence of N = 8 frames of a video of a moving escalator and their eigenimages are shown respectively in the upper and lower parts of Fig. 10.8.

It is interesting to observe that the first eigenimage corresponding to the greatest eigenvalue (left panel of the third row of the figure) represents mostly the static scene of the image frames representing the main variations in the image (carrying most of the energy), while the subsequent eigenimages represent mostly the motion in the video, the variation between the frames. For example, the motion of the people riding on the escalator is mostly reflected by the first few eigenimages following the first one, while the motion of the escalator stairs is mostly reflected in the subsequent eigenimages.

The 8×8 covariance matrix and the energy distribution among the eight components plot before and after the KLT are shown in Fig. 10.9

We see that due to the spatial correlation between nearby pixels, the covariance matrix before the KLT (left) can be modeled by the squared exponential function, while the covariance matrix after the KLT (middle) is completely decorrelated and the energy is highly compacted into a small number of principal components (here the first component), as also clearly shown in the comparison of the energy distribution before and after the KLT (right).

• Eigen-face and face recognition

The KLT is applied to a stack of twenty face images (N = 20) shown in Fig. 10.10 (credit to AT&T Laboratories Cambridge, downloadable from

Figure 10.8 Eight frames of a video before (top) and after (bottom) the KLT Figure 10.9 Covariance matrix before (left) and after (middle) the KLT, and energy redistribution (right)

https://cam-orl.co.uk/facedatabase.html), to generate the eigen-faces in Fig. 10.11.

The table below shows the percentage of energy contained in each of the

Figure 10.10 Face images

Figure 10.11 The eigen-faces

components:

components  1  2  3  4  5  6  7  8  9

percentage energy 48.5 11.6 6.1 4.6 3.8 3.7 2.6 2.5 1.9 accumulative energy 48.5 60.1 66.2 70.8 74.6 78.3 81.0 83.5 85.4

\begin{matrix}10 & 11 & 12 & 13 & 14 & 15 & 16 & 17 & 18 & 19 & 20 \\ 1.9 & 1.8 & 1.6 & 1.5 & 1.4 & 1.3 & 1.2 & 1.1 & 1.1 & 0.9 & 0.8 \\ 87.3 & 89. & 90.7 & 92.2 & 93.6 & 94.9 & 96.1 & 97.2 & 98.2 & 99.2 & 100.0 \end{matrix} (10.55)

The faces are then reconstructed based on 95% of the total information (15 out of 20 components) as shown in Fig. 10.12.

• Hand-written digit recognition

The goal here is to recognize hand-written digits from 0 to 9 given in image form of 16 × 16 pixels, in a dataset containing N = 224 × 10 samples for the K = 10 digits. A small subset of the samples is shown in Fig. 10.13. All 16 × 16 = 256 pixels in the image of each sample can be converted into a d = 256 vector by concatenating all columns (or rows) of the image. Then

Figure 10.12 Reconstructed faces Figure 10.13 Ten handwritten digits

the KLT can be carried out to reduce the dimensionality from d = 256 to some d′ ≪ d, based on either the covariance matrix Σ_{x} = S_{T} of all N = 2240 sample vectors representing the overall distribution of the dataset, or the between-class scatter matrix S_{B} previously considered representing the separability of the ten classes.

Specifically, we use the eigenvectors corresponding to the d′ greatest eigenvalues of the covariance matrix or the between-class scatter matrix to form a d′ by d transform matrix. After the KLT transform by the data, certain classification algorithm can be carried out in the much reduced d’ dimensional space.

The energy distribution over all d = 256 signal components is plotted in Fig. 10.14 for both the original signal before the KLT (top) and after the KLT based on both S_{B} (middle) and S_{T} (bottom).

For the KLT based on S_{T} with rank N − 1, d′ = 79 components are needed to keep 95.1% of the total energy, in comparison to the KTL based on S_{B} with rank rank(S_{B}) = K −1 = 9, requiring are only d′ = 9 principal components corresponding to the same number of non-zero eigenvalues of S_{B} to keep 100% of the total energy representing the separability information. The percentage energy contained in these non-zero eigenvalues are: 28.6%, 22.7%, 14.4%, 10.9%, 7.1%, 7.0%, 3.8%, 3.2%, 2.4%.

The d = 256 dimensional eigenvectors can be visualized when converted into 16 × 16 eigen-digits, representing the basis by which any original face images can be represented as a linear combination of such eigen-digits as shown in Fig. 10.15. We see that the 10th eigen-digit corresponding to a zero eigenvalue contains no meaningful information but random noise.

If we only keep the first two or three principal components (corresponding to the greatest eigenvalues) after the KLT, the dataset can be visualized as shown in Fig. 10.16. The sample points in each of the ten different classes are color-coded. It can be seen that even when the dimensions are much

Figure 10.14 Energy distribution before (top) and after the KLT based on S_{B} (middle), and S_{T} (bottom)

Figure 10.15 The first ten Eigen-digits

reduced from N = 256 to d = 3 or even d = 2, it is still possible to separate the ten classes reasonably well.

10.8 PCA for Feature Extraction

IN the PCA method considered in previous sections, the transform matrix is the eigenvector matrix of the covariance matrix of the dataset, of which the diagonal

Figure 10.16 Visualization of ten classes of digits

components are the variances of the signal components that measure the dynamic energy or information contained in all data points in the entire dataset.

In the context of classification, we can also use the PCA method to reduce the data dimensionality while maximally preserving the information of interest, which in this case should be pertinent to the separability of data points into different classes, instead of just the generic variation in all data points independent of their classes. Specifically, the transform matrix A = [a_{1}, ⋯, a_{d′}] in the linear transformation

y = A^{T} x  (10.56)

needs to be constructed based on certain measurement of the class separability, such as the within and between scatter matrices considered in Chapter 9, instead of the covariance matrix of the entire dataset.

• Optimal transformation for maximizing tr(S_{B})

We first consider the objective function J_{B}(A) in the new d’-D space as a function of the transform matrix A:

J(A) = tr (S_{B}^{y}) = tr (A^{T} S_{B}^{x} A) = tr [ a_{1}^{T} ; ⋮ ; a_{d′}^{T} ] S_{B}^{x}[a_{1}, ⋯, a_{d′}]

= tr [ a_{1}^{T} ; ⋮ ; a_{d′}^{T} ] [S_{B}^{x}a_{1}, ⋯, S_{B}^{x}a_{d′}] = ∑_{i=1}^{d′} a_{i}^{T} S_{B}^{x}a_{i}  (10.57)

To find optimal A that maximizes tr(S_{B}^{y}), we need to solve the following constrained maximization problem:

\{ \begin{matrix}maximize & J(A) = tr(A^{T} S_{B}^{x}A) = ∑_{i=1}^{d′} a_{i}^{T} S_{B}^{x}a_{i} \\ subject to & ||a_{j}||^{2} = a_{j}^{T} a_{j} = 1 (j = 1, ⋯, d′) \end{matrix}  (10.58)

where the constraint equations are to guarantee that the column vectors of A are normalized and therefore finite. This constrained optimization problem can be solved by Lagrange multiplier method. We first construct the Lagrangian function

L(A, λ) = J(A) − ∑_{j=1}^{d′} λ_{j}(a_{j}^{T} a_{j} − 1)  (10.59)

and then set its gradient with respect to each column vector a_{i} of A to zero:

\frac{∂}{∂a_{i}} L(A, λ) = \frac{∂}{∂a_{i}} [∑_{j=1}^{d′}(a_{j}^{T} S_{B}^{x}a_{j} − λ_{j}a_{j}^{T} a_{j} + λ_{j})]

= \frac{∂}{∂a_{i}} [a_{i}^{T} S_{B}^{x}a_{i} − λ_{i}a_{i}^{T} a_{i}] = 2S_{B}^{x}a_{i} − 2λ_{i}a_{i} = 0, i.e. S_{B}a_{i} = λ_{i}a_{i}
(i = 1, ⋯, n)  (10.60)

This happens to be the ith eigenequation of S_{B}, indicating the column vectors of the optimal A have to be the orthogonal eigenvectors {v_{1}, ⋯, v_{d′}} of the symmetric matrix S_{B}^{x}, i.e., A = [a_{1}, ⋯, a_{d′}] = V = [v_{1}, ⋯, v_{d′}]. Now the d′ new features can be obtained as

y = A^{T} x = [ v_{1}^{T} ; ⋮ ; v_{d′}^{T} ]_{d′×d} x_{d×1}, or y_{i} = v_{i}^{T} x, (i = 1, ⋯, d′) (10.61)

and

J(A) = ∑_{i=1}^{d′} v_{i}^{T} S_{B}v_{i} = ∑_{i=1}^{d′} v_{i}^{T} v_{i}λ_{i} = ∑_{i=1}^{d′} λ_{i} > 0  (10.62)

We note that as S_{B}^{x} is symmetric and positive definite, its eigenvalues are positive and its normalized eigenvectors are orthonormal: v_{i}^{T} v_{j} = δ_{ij}. To maximize J(A), the eigenvectors in A need to correspond to the d′ greatest eigenvalues of S_{B}.

• Optimal transformation for maximizing tr(S_{B/T})

The previous method only maximizes S_{B} without taking S_{W} or S_{T} = S_{B}+S_{W} into consideration. As S_{W}^{y} in the d’-D space may have also changed as well as S_{B}^{y}. To maximize the separability, we need to maximize S_{B} while at the same time also minimize S_{W}. Or, equivalently, we need to maximize S_{B}^{y} normalized by S_{T}^{y}:

S_{B/T}^{y} = (S_{T}^{y})^{−1} S_{B}^{y} = (A^{T} S_{T}^{x} A)^{−1}(A^{T} S_{B}^{x}A)
= A^{−1}(S_{T}^{x})^{−1}(A^{T})^{−1}A^{T} S_{B}^{x}A
= A^{−1}(S_{T}^{x})^{−1}S_{B}^{x}A = A^{−1}S_{B/T}^{x} A  (10.63)

We note that different from the symmetric matrix S_{B} considered above, matrix S_{B/T}^{y} as a product of two symmetric matrices is no longer symmetric, i.e., its eigenvectors are not orthogonal in general, and the method above cannot be used, we will find the optimal matrix A in some other way.

We first consider the special case of d′ = 1, i.e., A = a is an d × 1 vector and y = a^{T} x is a scalar. We desire to maximize the following objective function in a 1-D space:

(S_{T}^{y})^{−1} S_{B}^{y} = (a^{T} S_{T}^{x} a)^{−1}(a^{T} S_{B}^{x}a) = \frac{s_{B}^{y}}{s_{T}^{y}} = \frac{a^{T} S_{B}^{x}a}{a^{T} S_{T}^{x} a} = R(a) (10.64)

This function R(a) is the Rayleigh quotient (Section A.3.3) of the two symmetric matrices S_{B}^{x} and S_{T}^{x}. The optimal transform vector a that maximizes R(a) can be found by solving the corresponding generalized eigenvalue problem (Section A.3.2):

S_{B}^{x}v_{i} = λ_{i}S_{T}^{x} v_{i}, (i = 1, ⋯, d)  (10.65)

where λ_{i} = R(a) is an eigenvalue and v_{i} the corresponding eigenvector of S_{B/T} = (S_{T}^{x})^{−1}S_{B}^{x}. Obviously, the transform vector a that maximizes R(a) is the eigenvector corresponding to the greatest eigenvalue λ_{max} ≥ λ_{i} (i = 1, ⋯, d).

We next generalize the method above to the case of d′ > 1. The matrix form of the eigenequation above is:

S_{B}^{x}V = S_{T}^{x} VΛ  (10.66)

where Λ = diag(λ_{1}, ⋯, λ_{d}) is the diagonal eigenvalue matrix of (S_{T}^{x})^{−1}S_{B}^{x}, and V = [v_{1}, ⋯, v_{d}] the eigenvector matrix which is no longer orthogonal in general. This generalized eigenvalue problem can be solved by finding the matrix V that diagonalizes both S_{T}^{x} and S_{B}^{x} at the same time (same as how to get Eq. (A.137)):

\{ \begin{matrix}S_{B}^{y} = V^{T} S_{B}^{x}V = Λ \\ S_{T}^{y} = V^{T} S_{T}^{x} V = I \end{matrix}  (10.67)

Left multiplying (S_{T}^{y})^{−1} = I to the first, we get

S_{B/T}^{y} = (S_{T}^{y})^{−1}S_{B}^{y} = Λ  (10.68)

The optimal transform matrix is composed of the eigenvectors A = [v_{1}, ⋯, v_{d′}] corresponding to the d′ greatest of all d eigenvalues λ_{1} ≥ ⋯ ≥ λ_{d′} ≥ ⋯ ≥ λ_{d}, so that

– the signal components in y = [y_{1}, ⋯, y_{d′}]^{T} are completely decorrelated, each component y_{i} = v_{i}^{T} x carries certain separability information s_{B}^{(y_{i})}/s_{T}^{(y_{i})} = λ_{i} (i = 1, ⋯, d′), independent of others;

– the total separability contained in the d’-D space, as the sum of the d′ greatest eigenvalues ∑_{i=1}^{d′} λ_{i}, is maximized.

Problems

In this problem set and some later ones as well as many examples in future chapters, we will again use the handwritten digit dataset considered in Section 10.7. A small subset of the dataset is shown in Fig. 10.13, while the complete dataset containing 2240 handwritten digits can be downloaded from the Cambridge University Press website at www.[to come].org.

Also, an ASCII file containing a 256 × 2240 array X for all 2240 data samples in vector form and a 256-D vector y for their class labelings can be downloaded here: /e176/homework/data0to9.txt

This ASCII file can be read by the code below:

    data=load('data0to9.txt','data');  $ read ASCII file
    d=size(data,1)-1        % dimensionality of data
    N=size(data,2);         % total number of samples
    X=data(1:d,:);          % all N d-dimensional sample vectors
    y=data(d+1,:);          % and their labelings
    K=length(unique(y));    % number of classes
    for k=1:K               % for each of the K classes
        idx=find(y==k);     % indices of points in class k
        Xk=X(:,idx)         % collect all points in class k
    end

1. Carry out KLT for the 4-D iris dataset. Display the 4 × 4 covariance matrices before and after the transform, and verify (1) the trace (sum of all diagonal components of the variance matrix) is conserved, (2) all off-diagonal components become zero after the KLT, i.e., all signal components are completely decorrelated.

2. Visualize the 4-D iris dataset in both 2-D and 3-D spaces spanned respectively by the first two and three principal components. Color-code all 150 data points as red, green, or blue according to their class identities.

3. Carry out KLT for the handwritten digit dataset and observe the effects of signal decorrelation and energy redistribution of the transform. Display the 256 × 256 covariance matrices both before and after the transform in image form, same as in Figs. 10.5 and 10.9, and plot their diagonal components (representing the dynamic energy or information contained in each data component), same as in Fig. 10.14.

4. Visualize the handwritten digit dataset in both 2-D and 3-D spaces spanned respectively by the first two and three principal components. Color code all data points according to their class identities to appreciate how difficult it is to separate them into different classes. (That is what you need to do in the future.)

5. Find all pair-wise Bhattacharyya distances d_{B}(C_{i}, C_{j}), i, i = 2, ⋯, 10, j = 1, ⋯, i − 1} between any two of the ten classes in the handwritten digits dataset, and display them in a lower triangular distance matrix.

Note:

• As the 256-D covariance matrix Σ_{k} is estimated based on only 224 samples of class C_{k}, the rank of this estimated covariance matrix is no more than 224 − 1 = 223.

• As all 224 samples are for the same digit, they are similar to each other and highly correlated, the rank of Σ_{k} is likely to be much lower than 223.

We therefore see that the estimated Σ_{k} is non-invertible as its rank is much smaller than its dimension, and so is the sum of two such covariance matrices (needed in the Bhattacharyya distance). Consequently, the dimensionality of the dataset must be reduced significantly (d′ ≪ d = 256) to become smaller than the rank of Σ_{k} by the KLT transform, before the Bhattacharyya distances can be calculated.

6. Typically the KLT matrix is calculated as the eigenvector matrix of the covariance matrix Σ (same as the total scatter matrix S_{T}) of the entire dataset, and the resulting principal components reflect the signal variations of all data points in the dataset. Alternatively, the KLT can also be carried out based on the between-class scatter matrix S_{B} or S_{B/T} to maximize the trace (or equivalently based on the within-class scatter matrix S_{W} to minimize its trace), if the signal variation of interest is the separability of different classes in the context of classification.

Now carry out such a KLT to reduce the dimensionality of both the iris and handwritten digit datasets based on S_{B}. Visualize the datasets in both the 2-D and 3-D spaces spanned by the greatest principal components, with all data points color-coded according to their class identities.

7. As discussed in section 10.7, the method of PCA can be applied to image data based on the covariance matrices given in Eq. (10.7). Now carry out PCA to the face images, downloadable from https://cam-orl.co.uk/facedatabase.html). Observe the resulting eigen-faces to see how different types of variations among all faces are extracted by the eigen-faces corresponding to the first few greatest eigenvalues.

8. Repeat the previous problem based on the images of handwritten numbers. Observe the resulting eigen-digits.

11 Variations of PCA

In this chapter we consider a set of methods that are either some variations of the PCA method in the previous chapter or closely related to it to achieve similar effects for feature selection and extraction.

11.1 Kernel Methods

In the kernel method, all data points as a vector x = [x_{1}, ⋯, x_{d}]^{T} in the original d-dimensional feature space are mapped by a kernel function z = ϕ(x) into a higher, possibly infinite, dimensional space

x ⟹ z = ϕ(x)  (11.1)

The presumption of the method is that the data points only appear in the form of inner product x_{m}^{T}x_{n} in the algorithm, then based on the kernel trick, the kernel function z = ϕ(x) never needs to be actually carried out. In fact, the form of the kernel function ϕ(x) and the dimensionality of the higher dimensional space do not need to be explicitly specified or known.

The motivation for such a kernel mapping is that the relevant operations such as classification and clustering may be carried out much more effectively once the dataset is mapped to the higher dimensional space. For example, classes not linearly separable in the original d-dimensional feature space may be trivially separable in a higher dimensional space, as illustrated by the following examples.

Example 11.1 In 1-D space, two classes C_{−} = {x|(a ≤ x ≤ b)} and C_{+} = {x|(x ≤ a) or (x ≥ b)} shown in the first row of Fig. 11.1 are not linearly separable. However, by the following mapping from 1-D space to 2-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \end{matrix} ] = [ \begin{matrix}x \\ (x − (a + b)/2)^{2} \end{matrix} ]  (11.2)

the two classes can be separated by thresholding in the second dimension of the 2-D space.

Example 11.2 The method above can be generalized to spaces of higher dimensions. For example, the two classes in 2-D space shown in the middle row of Fig.

Figure 11.1 Examples of kernel mapping

11.1 are not linearly separable: C_{−} = {x, ||x|| < D} and C_{+} = {x, ||x|| > D}. However, by the following mapping from 2-D space to 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}^{2} + x_{2}^{2} \end{matrix} ]  (11.3)

the two classes can be trivially separated linearly by thresholding in the third dimension of the 3-D space.

Example 11.3 In 2-D space, in the exclusive OR dataset shown in the bottom row of Fig. 11.1 the two classes of C_{−} containing points in quadrants I and III, and C_{+} containing points in quadrants II and IV are not linearly separable. However, by mapping the data points to a 3-D space:

z = ϕ(x) = [ \begin{matrix}z_{1} \\ z_{2} \\ z_{3} \end{matrix} ] = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{1}x_{2} \end{matrix} ]  (11.4)

the two classes can be separated by simply thresholding in the third dimension of the 3-D space.

Definition: A kernel is a function that takes two vectors x_{m} and x_{n} as arguments and returns the inner product of their images z_{m} = ϕ(x_{m}) and z_{n} = ϕ(x_{n}):

K(x_{m}, x_{n}) = ϕ(x_{m})^{T} ϕ(x_{n}) = z_{m}^{T}z_{n}  (11.5)

The kernel function takes as input some two vectors x_{m} and x_{n} in the original space, and returns a scalar value as the inner product z_{m} and z_{n} in some higher dimensional space. If the data points in the original space only appear in the form of inner product, then the kernel function z = ϕ(x), called the kernel-induced implicit mapping, never needs to be explicitly specified, and the dimension of the new does not even need to be known.

The following is a set of commonly used kernel functions K(x, x′), which can be represented as an inner product of two vectors z = ϕ(x) and z′ = ϕ(x′) in a higher dimensional space.

• linear kernel

Given two vectors x and x′, the linear kernel is defined as the true inner product, without any kernel mapping:

K(x, x′) = x^{T} x′ = ∑_{i=1}^{d} x_{i}x′_{i}  (11.6)

• polynomial kernels The binomial theorem states:

(x + y)^{n} = ∑_{k=0}^{n} ( \begin{matrix}n \\ k \end{matrix} ) x^{n−k}y^{k} = ∑_{k=0}^{n} \frac{n!}{k!(n − k)!} x^{n−k}y^{k}  (11.7)

where the binomial coefficient

( \begin{matrix}n \\ k \end{matrix} ) = \frac{n!}{k!(n − k)!}  (11.8)

is the number of ways to distribute n items into two bins (k in one and n − k in the other). This result can be generalized to the multinomial case:

(x_{1} + ⋯ + x_{d})^{n} = ∑ ; ∑_{i=1}^{d} k_{i}=n ( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}

= ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!}x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}  (11.9)

where the multinomial coefficient

( \begin{matrix}n \\ k_{1}, ⋯, k_{d} \end{matrix} ) = \frac{n!}{k_{1}! ⋯ k_{d}!}  (11.10)

is the number of ways to distribute n balls into d bins with k_{i} balls in the ith bin, and the summation is over all possible ways to get d non-negative integers k_{1}, ⋯, k_{d} that add up to n.

Now consider the homogeneous polynomial kernel for d-dimensional vectors x = [x_{1}, ⋯, x_{d}]^{T} defined as

K(x, x′) = (x^{T} x′)^{n} = (x_{1}x′_{1} + ⋯ + x_{d}x′_{d})^{n}
= ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}})
= ϕ(x)^{T} ϕ(x′) = z^{T} z′  (11.11)

where

z = ϕ(x) = [\sqrt{\frac{n!}{k_{1}! ⋯ k_{d}!}} (x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}) , (k_{i} ≥ 0, ∑_{i=1}^{d} k_{i} = n)]^{T} (11.12)

In particular, when d = 2 and n = 2, the polynomial kernel defined over 2-D vectors x = [x_{1}, x_{2}]^{T} is:

K(x, x′) = (x^{T} x′)^{2} = (x_{1}x′_{1} + x_{2}x′_{2})^{2}
= (x_{1}x′_{1})^{2} + 2x_{1}x′_{1}x_{2}x′_{2} + (x_{2}x′_{2})^{2} = ϕ(x)^{T} ϕ(x′) = z^{T} z′ (11.13)

where z = ϕ(x) = [x_{1}^{2}, \sqrt{2}x_{1}x_{2}, x_{2}^{2}] is a mapping from x in 2-D space to z in 3-D space.

A non-homogeneous polynomial kernel is defined as

K(x, x′) = (1 + x^{T} x′)^{n}  (11.14)

• The radial basis function (RBF) kernel The RBF kernel is defined as

K(x, x′) = e^{−||x−x′||^{2}/2σ^{2}} = e^{−γ||x−x′||^{2}}  (11.15)

where γ = 1/2σ^{2} is a parameter that can be adjusted to fit each specific dataset. This kernel can be written as the inner product of two infinite dimensional vectors (for simplicity, we assume σ = 1):

K(x, x′) = e^{−||x−x′||^{2}/2} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} e^{x^{T} x′} = e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} \frac{(x^{T} x′)^{n}}{n!}

= e^{−||x||^{2}/2} e^{−||x′||^{2}/2} ∑_{n=0}^{∞} [\frac{1}{n!} ∑ ; ∑_{i=1}^{d} k_{i}=n \frac{n!}{k_{1}! ⋯ k_{d}!} ((x_{1}x′_{1})^{k_{1}} ⋯ (x_{d}x′_{d})^{k_{d}})]

= ∑_{n=0}^{∞} ∑ ; ∑_{i=1}^{d} k_{i}=n (e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}) (e^{−||x′||^{2}/2} \frac{x_{1}^{′k_{1}} ⋯ x_{d}^{′k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}})
= ϕ(x)^{T} ϕ(x′) = z^{T} z′  (11.16)

where

z = ϕ(x) = [e^{−||x||^{2}/2} \frac{x_{1}^{k_{1}} ⋯ x_{d}^{k_{d}}}{\sqrt{k_{1}! ⋯ k_{d}!}}, (n = 0, ⋯, ∞, ∑_{k=1}^{n} k_{i} = n)]^{T}

(11.17) is a vector in an infinite dimensional space. In particular, when d = 1 we have

K(x, x′) = e^{−(x−x′)^{2}/2} = e^{−x^{2}/2} e^{−x^{′2}/2} e^{xx′} = e^{−x^{2}/2} e^{−x^{′2}/2} ∑_{n=0}^{∞} \frac{(xx′)^{n}}{n!}

= ∑_{n=0}^{∞}(e^{−x^{2}/2} x^{n}/\sqrt{n!}) (e^{−x^{′2}/2} x^{′n}/\sqrt{n!})  (11.18)

where z = ϕ(x) = [e^{−x^{2}/2} x^{n}/\sqrt{n!}, (n = 0, ⋯, ∞)]^{T} is a kernel function that maps a 1-D space into an infinite dimensional space.

11.2 Kernel PCA

The PCA method maps the dataset to a lower dimensional space, and the kernel method maps the dataset to a higher dimensional space. These two methods are combined in kernel PCA to take advantage of separability in the high dimensional space and computational efficiency in the low dimensional space.

Specifically, we assume the given dataset X = [x_{1}, ⋯, x_{N}] in the original ddimensional space is mapped into Z = [z_{1}, ⋯, z_{N}] in a high dimensional space. Correspondingly, the inner product x_{m}^{T}x_{n} in the original space is mapped into an inner product in the high dimensional space by a kernel function:

z_{m}^{T}z_{n} = K(x_{m}, x_{n}) = k_{mn} (m, n = 1, ⋯, N)  (11.19)

The N by N kernel matrix can be found as:

K = Z^{T} Z = [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] [z_{1}, ⋯, z_{N}] = [ \begin{matrix}z_{1}^{T} z_{1} & ⋯ & z_{1}^{T} z_{N} \\ ⋮ & ... & ⋮ \\ z_{N}^{T}z_{1} & ⋯ & z_{N}^{T}z_{N} \end{matrix} ] = [ \begin{matrix}k_{11} & ⋯ & k_{1N} \\ ⋮ & ... & ⋮ \\ k_{N1} & ⋯ & k_{NN} \end{matrix} ]

(11.20)

To carry out KLT in the high dimensional space, we need to estimate the covariance matrix of z = ϕ(x):

Σ_{z} = \frac{1}{N} ∑_{n=1}^{N}(z_{n} − m_{z})(z_{n} − m_{z})^{T} = \frac{1}{N} ∑_{n=1}^{N} z_{n}z_{n}^{T} − m_{z}m_{z}^{T}  (11.21)

If we assume the mean vector of all N samples in Z = [z_{1}, ⋯, z_{N}] is zero m_{z} = 0 (to be justified later), Σ_{z} can be written as (same as Eq. (10.46)):

Σ_{z} = \frac{1}{N} ∑_{n=1}^{N}(z_{n}−m_{z})(z_{n}−m_{z})^{T} = \frac{1}{N} ∑_{n=1}^{N} z_{n}z_{n}^{T} = \frac{1}{N} [z_{1}, ⋯, z_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] = \frac{1}{N} ZZ^{T}

(11.22)

We further get its eigenequations:

Σ_{z}v_{n} = (\frac{1}{N} ZZ^{T} ) v_{n} = λ_{n}v_{n} (n = 1, 2, ⋯)  (11.23)

To express this equations in terms of the kernel matrix K = K^{T} K, we premultiply Z^{T} on both sides of the eigenequation above to get (same as Eq. (10.48)):

\frac{1}{N} Z^{T} Z (Z^{T} v_{n}) = \frac{1}{N} Ku_{n} = λ_{n} (Z^{T} v_{n}) = λ_{n}u_{n}  (11.24)

where we have defined u_{n} = Z^{T} v_{n}. Solving this eigenequation of the N × N matrix K/N we get its eigenvectors u_{n} corresponding to the eigenvalues λ_{n}, the same as the non-zero eigenvalues of Σ_{z} = ZZ^{T} /N. The eigenvectors v_{n} of Σ_{z} in Eq. (11.23) can now be written as:

v_{n} = \frac{1}{λ_{n}N} Z (Z^{T} v_{n}) = \frac{1}{λ_{n}N} Zu_{n}  (11.25)

which can be further normalized by imposing the condition ||v_{n}||^{2} = 1:

||v_{n}||^{2} = v_{n}^{T} v_{n} = \frac{1}{(λ_{n}N)^{2}} (Zu_{n})^{T} (Zu_{n}) = \frac{1}{(λ_{n}N)^{2}} u_{n}^{T} Z^{T} Zu_{n}
= \frac{1}{(λ_{n}N)^{2}} u_{n}^{T} Ku_{n} = \frac{1}{λ_{n}N} u_{n}^{T} u_{n} = \frac{1}{λ_{n}N} ||u_{n}||^{2} = 1  (11.26)

where we have used the fact Ku_{n} = λ_{n}Nu_{n} in Eq. (11.24). We can get the normalized eigenvectors of Σ_{z} in Eq. (11.23):

v_{n} = \frac{1}{\sqrt{λ_{n}N}} u_{n}  (11.27)

Now we can carry out KLT for any data point z = ϕ(x) in the high dimensional space by

y = V^{T} z = [v_{1}, ⋯, v_{N}]^{T} z  (11.28)

of which each component y_{n} = v_{n}^{T} z is the projection of z = ϕ(x) onto one of the eigenvectors v_{n} of Σ_{z} (n = 1, ⋯, N)

y_{n} = z^{T} v_{n} = z^{T} (\frac{1}{λ_{n}N} Zu_{n}) = \frac{1}{λ_{n}N} z^{T} [z_{1}, ⋯, z_{N}]u_{n}

= \frac{1}{λ_{n}N} [z^{T} z_{1}, ⋯, z^{T} z_{N}] u_{n} = \frac{1}{λ_{n}N} [k_{1}, ⋯, k_{N}] u_{n} = \frac{1}{λ_{n}N} k^{T} u_{n}(11.29)

where we have defined k = [k_{1}, ⋯, k_{N}]^{T} with k_{n} = z^{T} z_{n}. Although neither z nor v_{n} = Zu_{n}/λ_{n}N is explicitly available, their inner product y_{n} = z^{T} v_{n} can still be obtained based on the kernel function k and u. As alll data points in the high dimensional space appear only in the form of an inner product in the equation above, as well as in the eigenequation Eq. (11.24), the kernel mapping z = ϕ(x) never needs to be explicitly carried out.

The discussion above is based on the assumption that in the high dimensional space the data have zero mean m_{z} = 0. This cannot be achieved by subtracting the mean from the data to get z̃ = z−m_{z}, as neither z nor m_{z} is available without actually carrying out the kernel mapping z = ϕ(x). However, we can find the inner product of any two zero-mean data points z̃_{m}, z̃_{n}, (m, n = 1, ⋯, N) in the high dimensional space:

k̃_{mn} = z̃_{m}^{T}z̃_{n} = (z_{m} − m_{z})^{T} (z_{n} − m_{z}) = (z_{m} − \frac{1}{N} ∑_{k=1}^{N} z_{k})^{T} (z_{n} − \frac{1}{N} ∑_{l=1}^{N} z_{l})

= z_{m}^{T}z_{n} − z_{m}^{T} (\frac{1}{N} ∑_{l=1}^{N} z_{l}) − (\frac{1}{N} ∑_{k=1}^{N} z_{k}^{T} ) z_{n} + \frac{1}{N^{2}} ∑_{k=1}^{N} ∑_{l=1}^{N} z_{k}^{T} z_{l}

= k_{mn} − \frac{1}{N} ∑_{l=1}^{N} k_{ml} − \frac{1}{N} ∑_{k=1}^{N} k_{kn} + \frac{1}{N^{2}} ∑_{k=1}^{K} ∑_{l=1}^{N} k_{kl}  (11.30)

The kernel matrix composed of all N × N such inner products can be written as:

K̃ = K − 1_{N}K − K1_{N} + 1_{N}K1_{N}  (11.31)

where

1_{N} = \frac{1}{N} [ \begin{matrix}1 & ⋯ & 1 \\ ⋮ & ... & ⋮ \\ 1 & ⋯ & 1 \end{matrix} ]_{N×N}  (11.32)

In the algorithm considered above if we replace the kernel matrix K for z with non-zero mean by K̃ for z̃ with zero mean, then the assumption that z has zero mean m_{z} becomes valid. In summary, here are the steps of the kernel PCA algorithm:

• Given the dataset X = {x_{1}, ⋯, x_{N}}, obtain K based on kernel mapping k_{mn} = k(x_{m}, x_{n}) for all m, n = 1, ⋯, N, and then find K̃, as in Eq. (11.31);

• Solve eigenvalue problem of K̃ to get the eigenvalue λ_{n} and the corresponding eigenvector u_{n} for all i = 1, ⋯, N;

• Project each data point z = ϕ(x) (implicit) onto the nth eigenvector v_{n} to get the principal component: y_{n} = z^{T} v_{n} = k^{T} u_{n}/λ_{n}N for each n = 1, ⋯, N, as in Eq. (11.29).

• Carry out whatever operation (e.g., feature selection, classification) in the subspace spanned by the first M < N principal components {y_{1}, ⋯, y_{M}} in the high dimensional feature space, containing most of the information.

Note that in kernel PCA methods, the complexity for solving the eigenvalue problem of the N × N kernel matrix K is O(N^{3}), instead of O(d^{3}) for the linear PCA based on the d × d covariance matrix Σ_{x}.

Shown below is a Metlab function for the kernel PCA algorithm which takes the dataset X = [x_{1}, ⋯, x_{N}] as input and generates the eigenvalues in d and

Figure 11.2 Linear and kernel PCAs for a 2-D dataset of four classes

the dataset in the transform space Y. The function Kernel(X,X) calculates the kernel matrix K.

function [Y d]=KPCA(X)
[D N]=size(X);
K=Kernel(X,X);               % kernel matrix of data X
one=ones(N)/N;
Kt=K-one*K-K*one+one*K*one;  % Kernel matrix of data with zero mean
[U D]=eig(Kt);               % solve eigenvalue problem for K
[d idx]=sort(diag(D),'descend');
                             % sort eigenvalues in descending order
U=U(:,idx);
D=D(:,idx);
for n=1:N
    U(:,n)=U(:,n)/d(n)/N;
end
Y=(K*U)';
end

The linear and kernel PCAs (based on RBF kernel) are applied to three datasets and compared as shown in Figs. 11.2 through 11.4.

11.3 Factor Analysis and Expectation Maximization

The method of factor analysis (FA) models a set of d observed manifest variables in x = [x_{1}, ⋯, x_{d}]^{T} as a linear combination of a set of d′ < d unobserved hidden latent variables or common factors in z = [z_{1}, ⋯, z_{d′}]^{T}, to explain and reveal

Figure 11.3 Linear and kernel PCAs for the 256-D handwritten digits

the variability and dependency among the d observed variables, typically correlated, in terms of the latent variables, assumed to be independent and therefore uncorrelated. The method of FA is therefore similar to the method of PCA in the sense that they both aim at extracting the essential information contained in a multivariate dataset as some linear combination of all signal components and thereby reducing the dimensionality of the dataset while preserving most of the information contained in it.

Specifically, we assume each of the observed variables in x is a linear combination of the d′ factors in z

x_{i} = ∑_{j=1}^{d′} w_{ij}z_{j} + e_{i} = [w_{i1}, ⋯, w_{id′}] [ z_{1} ; ⋮ ; z_{d′} ] + e_{i},  (i = 1, ⋯, d) (11.33)

or in matrix form

x = [ x_{1} ; ⋮ ; ⋮ ; x_{d} ] = [ \begin{matrix}w_{11} & ⋯ & w_{1d′} \\ ⋮ & ... & ⋮ \\ ⋮ & ... & ⋮ \\ w_{d1} & ⋯ & w_{dd′} \end{matrix} ] [ z_{1} ; ⋮ ; z_{d′} ] + [ e_{1} ; ⋮ ; ⋮ ; e_{d} ] = Wz + e (11.34)

where W is a d × d′ factor loading matrix, and e_{i} is the noise associated with x_{i}. Also, for simplicity and without loss of generality, we assume the dataset has a zero mean. If W were available, the d′ factors in z can be found by solving this over-determined linear equation system of d equations but d′ < d unknowns by the least-squares method (with minimum squared error ||e||^{2}):

ẑ = W^{−}x = (W^{T} W)^{−1}W^{T} x  (11.35)

where W^{−} = (W^{T} W)^{−1}W^{T} is the left pseudo-inverse of W.

However, as W is unavailable, it needs to be estimated together with z at

Figure 11.4 Linear and kernel PCAs for the iris dataset

the same time, based on the given dataset X = [x_{1}, ⋯, x_{N}], typically N ≫ d. This can be done by the general method of expectation-maximization (EM), as discussed below.

Specifically, we treat both z and e as random vectors, and make the following assumptions:

• The latent variables in z are of zero mean, independent of each other, and with unity variance:

m_{z} = E[z] = 0,  Σ_{z} = Cov[z] = E[zz^{T}] = I  (11.36)

and they are normally distributed:

p(z) = N (0, I)  (11.37)

• The noise components in e have zero mean, and they are independent of each other with a diagonal covariance matrix:

m_{e} = E[e] = 0, Σ_{e} = Cov[e] = E[ee^{T}] = Ψ = [ \begin{matrix}ψ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & ψ_{d} \end{matrix} ]

(11.38)

and they are also normally distributed:

p(e) = N (0, Ψ)  (11.39)

• The latent variables and the noise are independent of each other:

Σ_{ze} = Cov[z, e] = E[(z − m_{z})(e − m_{e}^{T})] = E[ze^{T}] = 0  (11.40)

The two matrices W and Ψ defined above as the parameters of the FA model are denoted by θ = {W, Ψ}.

Based on the assumptions above, we desire to find the conditional pdf p(z|x) of the latent variable z given the observed variable x. To do so, we first find the pdf of x = Wz + e, which, as a linear combination of the two normally distributed random vectors z and e, is also normally distributed with p(x) = N (m_{x}, Σ_{x}), where

m_{x} = E[x] = E[Wz + e] = W E[z) + E(e] = 0  (11.41)
Σ_{x} = Cov[x] = E[(x − m_{x})(x − m_{x}^{T})] = E[xx^{T}] = E [(Wz + e)(Wz + e)^{T} ]
= W E[zz^{T}]W^{T} − E[ez^{T}]W^{T} − W E[ze^{T}] + E[ee^{T}]
= WW^{T} + Ψ  (11.42)

As both m_{x} and Σ_{x} are based on the model parameter θ = {W, Ψ}, the pdf of x is conditional on θ:

p(x|θ) = N (m_{x}, Σ_{x}) = N (0, WW^{T} + Ψ)  (11.43)

The joint distribution p(z, x) is also Gaussian with a zero mean

m = E ([ \begin{matrix}z \\ x \end{matrix} ]) = [ \begin{matrix}m_{z} \\ m_{x} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ]  (11.44)

and a covariance Σ composed of four submatrices:

Σ = [ \begin{matrix}Σ_{zz} & Σ_{zx} \\ Σ_{xz} & Σ_{xx} \end{matrix} ]  (11.45)

where

Σ_{zz} = I
Σ_{xx} = WW^{T} + Ψ
Σ_{zx} = Σ_{zx}^{T} = E[(z − m_{z})(x − m_{x})^{T}] = E[zx^{T}] = E[z(Wz + e)^{T}]
= E[zz^{T}]W^{T} + E[ze^{T}] = IW^{T} + 0 = W^{T}  (11.46)

The normal distribution p(z, x|θ) can now be expressed as:

p(x, z|θ) = p ([ \begin{matrix}z \\ x \end{matrix} ]) = N (m, Σ) = N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}Σ_{zz} & Σ_{zx} \\ Σ_{xz} & Σ_{xx} \end{matrix} ])
= N ([ \begin{matrix}0 \\ 0 \end{matrix} ] , [ \begin{matrix}I & W^{T} \\ W & WW^{T} + Ψ \end{matrix} ])  (11.47)

Based on this joint pdf of both x and z, we can further find the desired conditional pdfs of both p(x|z) and p(z|x) (Section B.1.5):

• p(x|z, θ) = N (m_{x|z}, Σ_{x|z}), with

\{ \begin{matrix}m_{x|z} & = & m_{x} + Σ_{xz}Σ_{zz}^{−1}(z − m_{z}) = Wz \\ Σ_{x|z} & = & Σ_{xx} − Σ_{xz}Σ_{zz}^{−1}Σ_{zx} = WW^{T} + Ψ − WW^{T} = Ψ \end{matrix} (11.48)

• p(z|x, θ) = N (m_{z|x}, Σ_{z|x}), with

\{ \begin{matrix}m_{z|x} & = & m_{z} + Σ_{zx}Σ_{xx}^{−1}(x − m_{x}) = W^{T} (WW^{T} + Ψ)^{−1}x = Bx \\ Σ_{z|x} & = & Σ_{zz} − Σ_{zx}Σ_{xx}^{−1}Σ_{xz} = I − W^{T} (WW^{T} + Ψ)^{−1}W = I − BW \end{matrix}

(11.49)

where we have defined

B = W^{T} (WW^{T} + Ψ)^{−1}  (11.50)

Note that while p(z) = N (0, I) has zero mean and diagonal covariance, the conditional distribution p(z|x, θ) = N (m_{z|x}, Σ_{z|x}) has non-zero mean m_{z|x} and non-diagonal covariance Σ_{z|x}.

The computational complexity for the inversion of the N ×N matrix WW^{T} + Ψ is O(N^{3}). However, by applying the Woodbury matrix identity (Section A.2.4):

(Ψ + WW^{T})^{−1} = Ψ^{−1} − Ψ^{−1}W(I + W^{T} Ψ^{−1}W)^{−1}W^{T} Ψ^{−1}  (11.51)

where Ψ as a diagonal matrix can be easily inverted, and I + W^{T} Ψ^{−1}W is an d′ × d′ matrix that can be inverted with complexity O(d^{′3}) ≪ O(N^{3}).

The model parameter θ = {W, Ψ} can now be estimated based on the given dataset X by the EM algorithm, an iterative process of the following two steps:

• The E-step:

Find the expectation of the log-likelihood function of the model parameter θ, to be maximized in the following M-step.

Find the likelihood function of θ based on the observed dataset X = [x_{1}, ⋯, x_{N}] (all samples assumed to be i.i.d.):

L(θ|X, z) = p(X, z|θ) = ∏_{n=1}^{N} p(x_{n}, z|θ) = ∏_{n=1}^{N} p(x_{n}|z, θ) p(z|θ) (11.52)

and the log-likelihood:

log L(θ|X, z) = ∑_{n=1}^{N} (log p(x_{n}|z, θ) + log p(z))  (11.53)

The second term can be dropped as p(z) = N (0, I) is independent of the model parameter θ and therefore irrelevant to the maximization of the log-likelihood with respect to θ.

We then find the expectation of the log-likelihood function, denoted by Q, with respect to the latent variable z based on Eq. (11.48):

Q = E_{z|x} [log L(θ|X, z)] = ∑_{n=1}^{N} E_{z|x_{n}} [log p(x_{n}|z, θ)]

= ∑_{n=1}^{N} E_{z|x_{n}} log [\frac{1}{(2π)^{d/2}|Σ_{x|z}|^{1/2}} exp (−\frac{1}{2}(x_{n} − m_{x|z})^{T} Σ_{x|z}^{−1}(x_{n} − m_{x|z}))]

= ∑_{n=1}^{N} E_{z|x_{n}} [−\frac{d}{2} log(2π) − \frac{1}{2} log |Ψ| − \frac{1}{2}(x_{n} − Wz)^{T} Ψ^{−1}(x_{n} − Wz)]

= −\frac{dN}{2} log(2π) − \frac{N}{2} log |Ψ| − \frac{1}{2} ∑_{n=1}^{N} E_{z|x_{n}} [(x_{n} − Wz)^{T} Ψ^{−1}(x_{n} − Wz)]

= C − \frac{N}{2} log |Ψ| − \frac{1}{2} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} Ψ^{−1}x_{n} − 2x_{n}^{T} Ψ^{−1}Wz + z^{T} W^{T} Ψ^{−1}Wz] (11.54)

where we have used the fact that

x_{n}^{T} Ψ^{−1}(Wz) = (Wz)^{T} Ψ^{−1}x_{n}  (11.55)

as both Ψ^{−1} and Ψ are diagonal and therefore also symmetric. Here the constant C = −dN log(2π)/2 can be dropped.

• The M-step:

Find the optimal model parameter θ = {W, Ψ} that maximizes the expectation of the log-likelihood Q obtained in the E-step.

We set to zero the derivative of Q with respective each of the two parameters in θ = {W, Ψ} and solve the resulting equations.

– Find W:

\frac{∂Q}{∂W} = \frac{∂}{∂W} ∑_{n=1}^{N} E_{z|x_{n}} [−x_{n}^{T} Ψ^{−1}x_{n} + 2x_{n}^{T} Ψ^{−1}Wz − z^{T} W^{T} Ψ^{−1}Wz]

= ∑_{n=1}^{N} E_{z|x_{n}} [\frac{∂}{∂W} (2x_{n}^{T} Ψ^{−1}Wz − z^{T} W^{T} Ψ^{−1}Wz)]

= ∑_{n=1}^{N} E_{z|x_{n}} [2Ψ^{−1}x_{n}z^{T} − 2Ψ^{−1}Wzz^{T} ]

= 2Ψ^{−1} ∑_{n=1}^{N} (x_{n}E_{z|x_{n}}[z^{T}] − WE_{z|x_{n}}[zz^{T}]) = 0  (11.56)

Solving for W we get

W = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z^{T}]) (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}])^{−1}  (11.57)

where E_{z|x_{n}}[z] and E_{z|x_{n}}[zz^{T}] can be found in Eq. (11.49):

\{ \begin{matrix}E_{z|x_{n}}[z] & = & m_{z|x_{n}} = Bx_{n} \\ E_{z|x_{n}}[zz^{T}] & = & Σ_{z|x_{n}} + m_{z|x_{n}}m_{z|x_{n}}^{T} = I − BW + Bx_{n}x_{n}^{T} B^{T} \end{matrix}

(11.58)

The second equation is due to the fact that Σ_{z} = E[zz^{T}] − m_{z}m_{z}^{T}. Here E_{z|x_{n}}[z] can be considered as the estimation of the z, while E_{z|x_{n}}[zz^{T}] the uncertainty of the estimation.

– Find Ψ:

\frac{∂Q}{∂Ψ^{−1}} = \frac{∂}{∂Ψ^{−1}} [N log |Ψ| + ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} Ψ^{−1}x_{n} − 2x_{n}^{T} Ψ^{−1}Wz + z^{T} W^{T} Ψ^{−1}Wz]]

= −NΨ + ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}x_{n}^{T} − 2x_{n}z^{T} W^{T} + W(zz^{T})W^{T} ]

= −NΨ + ∑_{n=1}^{N} x_{n}x_{n}^{T} − 2 ∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} W^{T} + W ∑_{n=1}^{N} E_{z|x}[zz^{T}]W^{T} = 0(11.59)

Solving for Ψ we get

Ψ = \frac{1}{N} diag (∑_{n=1}^{N} x_{n}x_{n}^{T} − 2 (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} ) W^{T} + W (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}]) W^{T} )

(11.60)

where diag(A) is the operation that sets all off-diagonal elements of matrix A to zero, so that the resulting matrix is indeed diagonal as what Ψ should be. We further replace W in front of the last term above by that in Eq. (11.57) and get

Ψ = \frac{1}{N} diag (∑_{n=1}^{N} x_{n}x_{n}^{T} − ∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z]^{T} W^{T} )  (11.61)

In summary, here are the steps of the EM algorithm:

1. Initialize parameters θ_{old} = {W_{old}, Ψ_{old}};

2. E-step:

Find E_{z|x}(z) and E_{z|x}(zz^{T}) in Eq. (11.58) based on m_{z|x_{n}} and Σ_{z|x_{n}} in Eq. (11.49), which in turn is based on θ_{old} = {W_{old}, Ψ_{old}};

3. M-step:

Find θ_{new} = {W_{new}, Ψ_{new}} in Eqs. (11.57) and (11.61), based on E_{z|x}(z) and E_{z|x}(zz^{T});

4. Terminate if convergence criterion is satisfied, otherwise replace θ_{old} by θ_{new} and return to the E-step.

As the E-step and M-step of are interdependent on each other, they need to be carried out iteratively based on certain initial guess of the parameters in θ. In general, the EM algorithm is an iterative method for getting the maximum likelihood (ML) or maximum a posteriori (MAP) estimates of the parameters in a statistical model, based on some unobserved latent variables. The EM algorithm is widely used in machine learning and data science, such as in the method of probabilistic PCA considered in the following section, and the Gaussian mixture model for clustering analysis to be considered in Section 15.2.

11.4 Probabilistic PCA

The method of probabilistic PCA (PPCA) can be considered as a special case of factor analysis based on the model x = Wz + e, when the random noise e is assumed to have an isotropic normal distributions with covariance matrix Ψ = εI. More specially, if we further let ε → 0 approach zero, the iteration of the EM algorithm for the PPCA becomes extremely simple, as we will see below.

Same as in Eqs. (11.37) and (11.39) for FA, here we also have the normal distribution of both z and e:

p(z) = N (0, I),  p(e) = N (0, εI)  (11.62)

and the normal distribution of the observed data x = Wz + e, same as in Eq. (11.43):

p(x) = N (0, WW^{T} + εI)  (11.63)

The goal here is to estimate ε and W as the model parameter, based on the observed data dataset X = [x_{1}, ⋯, x_{N}].

Now Eqs. (11.49) through (11.51) are rewritten as:

\{ \begin{matrix}m_{z|x_{n}} = E_{z|x_{n}}(z) = Bx_{n} \\ Σ_{z|x_{n}} = Cov_{z|x_{n}}(z) = I − BW \end{matrix}  (11.64)
B = W^{T} (WW^{T} + εI)^{−1}  (11.65)
(WW^{T} + εI)^{−1} = ε^{−1}I − ε^{−1}W(εI + W^{T} W)^{−1}W^{T}  (11.66)

The two steps of the EM algorithm become:

• E-step: Get the expectation of the log-likelihood by replacing Ψ in Eq. (11.54) by εI:

Q = −\frac{dN}{2} log ε − \frac{1}{2ε} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} x_{n} − 2x_{n}^{T} Wz + z^{T} W^{T} Wz] (11.67)

• M-step: Find the optimal model parameters W and ε that maximize the expectation of the log-likelihood Q above, by setting the derivative of Q with respect to the parameters to zero and solve the resulting equations. For W, the result is the same as Eq. (11.57):

W_{new} = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}[z^{T}]) (∑_{n=1}^{N} E_{z|x_{n}}[zz^{T}])^{−1}  (11.68)

where

\{ \begin{matrix}E_{z|x_{n}}[z] & = & m_{z|x_{n}} = Bx_{n} \\ E_{z|x_{n}}[zz^{T}] & = & Σ_{z|x_{n}} + m_{z|x_{n}}m_{z|x_{n}}^{T} \end{matrix}  (11.69)

To find ε, we solve the equation

\frac{∂Q}{∂ε} = −\frac{dN}{2ε} + \frac{1}{2ε^{2}} ∑_{n=1}^{N} E_{z|x_{n}} [x_{n}^{T} x_{n} − 2x_{n}^{T} Wz + z^{T} W^{T} Wz] = 0 (11.70)

and get

ε = \frac{1}{dN} ∑_{n=1}^{N} [x_{n}^{T} x_{n} − 2x_{n}^{T} WE_{z|x_{n}}(z) + tr(W^{T} E_{z|x_{n}}(zz^{T})W)]

= \frac{1}{dN} ∑_{n=1}^{N} [||x_{n}||^{2} − 2E_{z|x_{n}}(z^{T})W^{T} x_{n} + tr(E_{z|x_{n}}(zz^{T})W^{T} W)] (11.71)

where we have used the fact that a^{T} b = tr(ab^{T}).

In summary, here are the steps in the EM algorithm for PPCA:

1. Initialize parameters θ_{old} = {W_{old}, ε_{old}};

2. E-step: Based on θ_{old}, find m_{z|x_{n}} and Σ_{z|x_{n}} in Eq. (11.64), and E_{z|x_{n}}(z) and E_{z|x_{n}}(zz^{T}) in Eq. (11.69);

3. M-step: Find θ_{new} = {W_{new}, ε_{new}} by evaluating Eqs. (11.68) and (11.71);

4. Terminate if convergence criterion is satisfied, otherwise replace θ_{old} by θ_{new} and return to step 2.

Specially, if we further assume ε → 0 and the latent variables in z are deterministic, i.e., E_{z|x}(z) = z. At the limit where ε = 0, x is simply a linear combination of the latent variables in z:

x = Wz + εI ⟹^{ε→0} Wz = [w_{1}, ⋯, w_{d′}] [ z_{1} ; ⋮ ; z_{d′} ] = ∑_{i=1}^{d′} z_{i}w_{i}  (11.72)

Comparing this expression of x with Eq. (10.45)

x = V′y = ∑_{i=1}^{d′} y_{i}v_{i}  (11.73)

we see that the column vectors in W = [w_{1}, ⋯, w_{d′}] for the PPCA are similar to the column vectors of V , the eigenvector matrix of Σ_{x}, in the linear PCA, as both of them can be viewed as the principal directions, an alternative set of basis vectors that also span the same d-dimensional feature space in which x resides. When x is expressed as a linear combination of these basis vectors, it can be approximated by only d′ < d of the d such basis vectors weighted by the d′ principal components in y = [y_{1}, ⋯, y_{d′}]^{T} in linear PCA, or the d′ factors in z = [z_{1}, ⋯, z_{d′}]^{T} in PPCA.

Here are the two EM steps for the PPCA:

• E-step: If ε → 0, Eq. (11.64) can be written as:

m_{z|x_{n}} = E_{z|x_{n}}(z) = z_{n} = Bx_{n} = W^{T} (WW^{T} + εI)^{−1}x_{n}
⟹^{ε→0} W^{T} (WW^{T})^{−1}x_{n} (n = 1, ⋯, N)  (11.74)

All N such equations can be written in matrix form as

Z = [z_{1}, ⋯, z_{n}] = W^{T} (WW^{T})^{−1}[x_{1}, ⋯, x_{N}] = W^{T} (WW^{T})^{−1}X

(11.75)

As W is a d × d′ matrix (d′ < d), the rank of the d × d matrix WW^{T} is at most d′, i.e., the inverse of WW^{T} does not exist and we cannot find z_{n} = W^{T} (WW^{T})^{−1}x_{n}. In fact the transformation matrix W^{T} (WW^{T})^{−1} is the right pseudo inverse (Section A.6.1) of the d × d′ matrix W, which exists only if d′ ≥ d.

However, we note that the model x = Wz is an over-constrained linear system of d equations but only d′ < d variables in z = [z_{1}, ⋯, z_{d′}]^{T}, and the least-squares solution that minimizes the squared error ||x−Wz||^{2} can be found by the left pseudo inverse (Section A.6.1) of W:

z_{n} = (W^{T} W)^{−1}W^{T} x_{n} = W^{−}x_{n} (n = 1, ⋯, N)  (11.76)

and we can get the latent variables in Z = [z_{1}, ⋯, z_{N}] given all N data points in X = [x_{1}, ⋯, x_{N}]:

Z = (W^{T} W)^{−1}W^{T} X = W^{−}X  (11.77)

which is based on the old value of W previously estimated.

• M-step: Now Eq. (11.68) can be written as:

W = (∑_{n=1}^{N} x_{n}E_{z|x_{n}}(z^{T})) (∑_{n=1}^{N} E_{z|x_{n}}(zz^{T}))^{−1}

= [x_{1}, ⋯, x_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ] ([z_{1}, ⋯, z_{N}] [ z_{1}^{T} ; ⋮ ; z_{N}^{T} ])^{−1}

= XZ^{T} (ZZ^{T})^{−1} = XZ^{−}  (11.78)

by which W is updated based on Z found previously in the E-step. We note that Z^{T} (ZZ^{T})^{−1} is the right pseudo inverse of the d × N matrix Z, which exists if d ≤ N.

Again, as the E-step and M-steps are interdependent on each other, they need to be carried out iteratively:

E-step:  Z = (W^{T} W)^{−1}W^{T} X = W^{−}X
M-Step:  W = XZ^{T} (ZZ^{T})^{−1} = XZ^{−}  (11.79)

We note that given the dataset X, the FA model x = Wz can be expressed as an over-determined matrix equation

X_{d×N} = [x_{1}, ⋯, x_{N}] = W_{d×d′}[z_{1}, ⋯, z_{N}] = W_{d×d′}Z_{d′×N}  (11.80)

or, taking transpose of the equation, we get

X_{N×d}^{T} = Z_{N×d′}^{T}W_{d′×d}^{T}  (11.81)

Given X, this equation can be interpreted in two ways:

• If W is available, this is a system of d equations but d′ < d unknowns in each of the N columns of Z, which is solved as in the E-step above;

• If Z is available, this is a system of N equations but d′ < N unknowns in each of the d rows of W, which is solved as in the M-step above.

In either step, the pseudo inverse method can be used to minimize the squared error ||X − WZ||^{2}.

When compared to the regular PCA method that finds all d eigenvalues and the corresponding eigenvectors all at once by solving the eigenequation Σ_{x}w = λw, the PPCA method only finds the d′ column vectors of matrix W as the basis vectors, as the basis vectors that span a subspace with much reduced dimensionality but containing the most essential information in the data. However, unlike the PCA, the basis vectors in PPCA are not necessarily orthogonal to each other.

The implementation of the PPCA is extremely simple, as shown in the Metlab code below. Based on some initialized W, the EM iteration is composed of the E-step and M-step:

    while er>tol
        Z=inv(W'*W)*W'*X;       % find Z given W in E-step
        Wnew=X*Z'*inv(Z*Z');    % find W given Z in M-step
        er=norm(Wnew-W);        % the LS error
        W=Wnew;
    end

Figs. 11.5 and 11.6 show the first few iterations of the PPCA based on the EM method presented above, for a set of data points in X = [x_{1}, ⋯, x_{N}] in 2-D and 3-D. The straight lines in the plots represents the column vectors of W, which is randomly initialized, but quickly approach the directions corresponding to the principal components obtained by the PCA method, along which the data points are most widely spread.

Figure 11.5 Column vectors in W iteratively approach the principal directions of dataset (2-D)

Figure 11.6 Column vectors in W iteratively approach the principal directions of dataset (3-D)

Fig. 11.7 shows the dataset of 10 hand-written numbers from 0 to 9, containing 2240 data points in a 256-D space, but now linearly mapped to a 3-D space spanned by d′ = 3 factors found by the PPCA method.

Figure 11.7 First three components of the handwritten digits dataset by the PPCA

11.5 Classical Multidimensional Scaling

The goal of multidimensional scaling (MDS) is to reduce the dimensionality of the dataset representing a set of N objects of interest, each described by a set of d features and represented as a vector x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional space, while the pairwise similarity relationship between any two of these objects are preserved. MDS can be used to visualize datasets in a high dimensional space. Specially, here we consider the classical MDS (cMDS), also known as principal coordinates analysis (PCoA), as one of the methods in MDS.

Specifically, we represent the pairwise similarity of a set of N objects X = [x_{1}, ⋯, x_{N}] by the N × N similarity matrix D_{x} = [ d_{ij}^{x} ], of which the ij-th component d_{ij}^{x} is certain measurement of the similarity between x_{i} and x_{j}. The goal is to map the dataset X in the d-dimensional space into Y = [y_{1}, ⋯, y_{N}] in a lower d’-dimensional space, in which the similarity matrix D_{x} is optimally approximated by D_{y} = [ d_{ij}^{y}]. When d′ ≤ 3, these data points in Y can be visualized. MDS can be considered as a method for dimension reduction. In some cases only the pairwise similarities d_{ij}^{x} (i, j = 1, ⋯, N) are given, while the N objects in X are not explicitly given, and the dimensionality d may be unknown.

We assume the given pairwise similarity is the Euclidean distance d_{ij} = ||x_{i} − x_{j}||_{2} between x_{i} and x_{j}:

d_{ij}^{2} = ||x_{i} − x_{j}||^{2} = (x_{i} − x_{j})^{T} (x_{i} − x_{j}) = x_{i}^{T} x_{i} − 2x_{i}^{T} x_{j} + x_{j}^{T} x_{j} (11.82)

and get

D_{x}^{2} = [ d_{ij}^{2}  ]_{N×N} = [ x_{i}^{T} x_{i} − 2x_{i}^{T} x_{j} + x_{j}^{T} x_{j}  ]_{N×N} = X_{r}−2X^{T} X+X_{c}

(11.83)

where

X_{r} = [ \begin{matrix}||x_{1}||^{2} & ⋯ & ||x_{1}||^{2} \\ ⋮ & ... & ⋮ \\ ||x_{N}||^{2} & ⋯ & ||x_{N}||^{2} \end{matrix} ] = X_{c}^{T}

X^{T} X = [ x_{1}^{T} ; ⋮ ; x_{N}^{T} ] [x_{1}, ⋯, x_{N}] = [ \begin{matrix}x_{1}^{T} x_{1} & ⋯ & x_{1}^{T} x_{N} \\ ⋮ & ... & ⋮ \\ x_{N}^{T}x_{1} & ⋯ & x_{N}^{T}x_{N} \end{matrix} ] (11.84)

Our goal is to find Y = [y_{1}, ⋯, y_{N}] in a lower dimensional space so that its pairwise similarity matrix D_{y} matches D_{x} optimally, in the sense that the following objective function is minimized:

J(Y) = ||D_{y}^{2} − D_{x}^{2}||  (11.85)

We note that such a solution Y is not unique, as any translated version of Y has the same pairwise similarity matrix and is also a solution. We therefore seek to find a unique solution Y in which all data points are centralized, i.e, the mean of all points in Y is zero:

ȳ = \frac{1}{N} ∑_{n=1}^{N} y_{n} = 0  (11.86)

To do so, we introduce an N × N symmetric centering matrix C = I − 1/N = C^{T} (Section A.8.1), where1 = ee^{T} is an N × N matrix with all components equal 1. Premultiplying C to a column vector a = [a_{1}, ⋯, a_{N}]^{T} removes the mean ā = ∑_{i=1}^{N} a_{i}/N of a from each of its component:

Ca = Ia−\frac{1}{N} 1a = [ \begin{matrix}1 & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 1 \end{matrix} ] [ a_{1} ; ⋮ ; a_{N} ]−\frac{1}{N} [ \begin{matrix}1 & ⋯ & 1 \\ ⋮ & ... & ⋮ \\ 1 & ⋯ & 1 \end{matrix} ] [ a_{1} ; ⋮ ; a_{N} ] = [ a_{1} − ā ; ⋮ ; a_{N} − ā ]

(11.87)

Taking transpose of the equation above, we get

(Ca)^{T} = a^{T} C = [a_{1} − ā, ⋯, a_{N} − ā]  (11.88)

i.e., postmultiplying C to a row vector a^{T} = [a_{1}, ⋯, a_{N}] removes the mean ā from each of its component.

We can now postmultiply C to X to remove the mean of each row of X:

XC = [x_{1}, ⋯, x_{N}]C = [x̄_{1}, ⋯, x̄_{N}] = X̄  (11.89)

i.e., the nth column becomes:

x̄_{n} = x_{n} − x̄, where x̄ = \frac{1}{N} ∑_{n=1}^{N} x_{n}  (11.90)

and premultiply C to X^{T} to remove the mean of each column of X^{T}:

CX^{T} = X̄^{T}  (11.91)

We further apply double centering to D_{x}^{2} by pre and post-multiplying C and get:

CD_{x}^{2}C = C (X_{r} − 2X^{T} X + X_{c}) C = CX_{r}C − 2C (X^{T} X) C + CX_{c}C
= −2C (X^{T} X) C = −2 (XC)^{T} (XC) = −2X̄^{T} X̄

Note that the first term CX_{r}C = C(X_{r}C) = C0 = 0, as all components of each row of X_{r} are the same as their mean, and removing the mean results in a zero row vector, X_{r}C = 0; and the last term CX_{c}C = (CX_{c})C = 0C = 0, as all components of each column of X_{c} are the same as their mean, and removing the mean results in a zero column vector.

Now the similarity matrix D_{x}^{2} of all centralized data points in X̄ with zero mean can be represented by the Gram matrix of X̄:

Gx̄ = X̄^{T} X̄ = −\frac{1}{2}CD_{x}^{2}C  (11.92)

We further assume all data points in Y are also centered with zero mean and their corresponding similarity matrix is also represented by Gȳ, then the objective function can be expressed as:

J(Y) = ||Gȳ − Gx̄|| = ||Ȳ^{T} Ȳ − Gx̄||  (11.93)

To find Ȳ that minimizes J(Y), we first consider the equation Ȳ^{T} Ȳ − Gx̄ = 0 so that J(Y) = 0. In other words, we desire to express Gx̄ as the inner product of some matrix with itself. To do so, we carry out eigenvalue decomposition of G_{x} to find its eigenvalue matrix Λ and the orthogonal eigenvector matrix V satisfying G_{x}V = VΛ, i.e.,

Gx̄ = VΛV^{−1} = VΛV^{T} = [v_{1}, ⋯, v_{N}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{N} \end{matrix} ] [ v_{1}^{T} ; ⋮ ; v_{N}^{T} ]

= (VΛ^{1/2}) (Λ^{1/2}V^{T} ) = (Λ^{1/2}V^{T} )^{T} (Λ^{1/2}V^{T} )

Note that as Gx̄ is symmetric, the eigenvalues are real and the eigenvectors are orthogonal, i.e., V^{T} = V^{1} is is orthogonal matrix. Now we can find the desired Y as:

Y = Λ^{1/2}V^{T} = [ \begin{matrix}\sqrt{λ_{1}} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & \sqrt{λ_{N}} \end{matrix} ] [ v_{1}^{T} ; ⋮ ; v_{N}^{T} ] = [ \sqrt{λ_{1}}v_{1}^{T} ; ⋮ ; \sqrt{λ_{N}}v_{N}^{T} ] (11.94)

Each column y_{i} in Y = [y_{1}, ⋯, y_{N}] is an N-D vector. If all N dimensions of these y_{n} are used, Y^{T} Y = G_{x} and J(Y = 0. To reduce the dimensionality to d′ < d, we construct Y′ by the first d′ rows of Y corresponding to the d′ greatest eigenvalues of λ_{1} ≥ ⋯ ≥ λ_{d′} ≥ ⋯ ≥ λ_{N} and their corresponding eigenvectors v_{1}, ⋯, v_{d′}. The error is:

J(Y′) = ||Y^{′T} Y′ − Y^{T} Y|| = || ∑_{i=1}^{d′} λ_{i}v_{i}v^{T} − ∑_{i=1}^{N} λ_{i}v_{i}v^{T} || = || ∑_{i=d′}^{N} λ_{i}v_{i}v^{T} ||

(11.95)

In summary, here is the PCoA algorithm:

• Given the pairwise similarity, an N × N matrix D_{x} = [ d_{ij} ], construct the squared proximity matrix D_{x}^{2} = [ d_{ij}^{2} ];

• Apply double centering to get G_{x} = −CD_{x}^{2}C/2;

• Find the d′ greatest eigenvalues λ_{1}, ⋯, λ_{d′} of G_{x}, and the corresponding eigenvectors v_{1}, ⋯, v_{d′};

• Get the map

Y_{d′×N} = Λ_{d′}^{1/2}V_{d′} = [ \sqrt{λ_{1}}v_{1}^{T} ; ⋮ ; \sqrt{λ_{d′}}v_{d′}^{T} ]  (11.96)

The Metlab functions for the PCA and PCoA are listed below:

function Y=PCA(X)
[V D]=eig(cov(X'));        % solve eigenequation of covariance
[d idx]=sort(diag(D),'descend');
                           % sort eigenvalues in descend order
V=V(:,idx);                % reorder corresponding eigenvectors
Y=V'*X;                    % carry out KLT
end
function Y=PCoA(X)
N=size(X,2);
D=zeros(N);                % pairwise similarity matrix
C=eye(N)-ones(N)/N;        % centering matrix
for i=1:N
    for j=1:N
        D(i,j)=(norm(X(:,i)-X(:,j)))^2;
    end
end
B=-C*D*C/2;
[V D]=eig(B);              % solve eigenequation for matrix B
[d idx]=sort(diag(D),'descend');
                           % sort eigenvalues in descend order
V=V(:,idx);                % reorder eigenvectors

Figure 11.8 Reconstruction of the locations of north America cities by the MDS

    Y=sqrt(D)*V';              % carry out PCoA
end

In Fig. 11.8, the top panel shows the locations of some major cities in North America, and the bottom panel shows the reconstructed city locations by the MDS method based on the pairwise distance of these cities. Note that the reconstructed map is a rotated and centralized version of the original map.

The methods of PCA and PCoA are compared when applied to the 4-D iris dataset and the 256-D handwritten digits dataset, with their corresponding visualizations in both 2-D and 3-D shown in Figs. 11.9 and 11.10.

Figure 11.9 Visualization of the iris dataset based on both PCA and PCoA

11.6 t-Distributed Stochastic Neighbor Embedding

The method of PCA considered previously is often used to reduce the dimensionality of a given dataset X = [x_{1}, ⋯, x_{N}] by an orthogonal mapping from the high dimensional space to a low dimensional space. However, sometimes this method based on the linear KLT transform may not be able to properly interpret complex nonlinear relationship between the variables that spanned the feature space, as it may not be able to maintain the relative pairwise distances between the data points in the resulting lower dimensional space. For example, in the low dimensional space after the transform, those dissimilar data points far apart from each other in a large scale in the original high dimensional space may be well represented in the low dimensional space by the major principal components along the axes with large variances, but the differences between those points dissimilar with each other in some smaller scales may not be properly represented, or even completely neglected, if their dissimilarity is mostly represented in the dimensions with small variances. In other words, the PCA method tends to emphasize large scale global structures of the dataset, while ignoring the small scale local structures, which may be equally important or even of greater interest, and therefore also need to be properly represented.

Also, when the dimensionality is much reduced from a high-D space to a low-D space, the crowding problem occurs. In a high-D space, a data point may have a large number of similar neighbors of roughly equal distance; however, in a low-D space, the number of equal-distance neighbors is significantly reduced (e.g., 8

Figure 11.10 Visualization of the handwritten digits dataset based on both PCA and PCoA

immediate neighbors in a 3-D space, 4 in a 2-D space, 2 in a 1-D space). Thus the space available for the neighbors of a point is reduced while the dimensionality is reduced. As the result, when a large number of neighbors in the high-D are mapped to a low-D space, they are forced to spread out, some may get closer to those that are originally farther away in the high-D space, thereby reducing the gaps between potential clusters of similar points.

In contrast, the t-Distributed Stochastic Neighbor Embedding (tSNE) method as a nonlinear method that is based on probability distributions of the data points being neighbors, and it attempts to preserve the structure at all scales, but emphasizing more on the small scale structures, by mapping nearby points in high-D space to nearby points in low-D space.

In the high-D space, the similarity between any data point x_{i} and each of other points x_{j} in the dataset X is defined based on the Gaussian kernel as a conditional probability:

p_{j/i} = \frac{exp(−||x_{i} − x_{j}||^{2}/2σ_{i}^{2})}{∑_{k=1}^{N} ∑_{l=1}^{N} exp(−||x_{k} − x_{l}||^{2}/2σ_{i}^{2})}  (11.97)

where σ_{i}^{2} is the variance of the data points in the local neighborhood of x_{i}.

We see that if any x_{j} is close to x_{i}, ||x_{i} − x_{j}|| is small and p_{j/i} is large, indicating x_{j} is similar to x_{i}; but if x_{j} is far away from x_{j}, ||x_{i} −x_{j}|| is large and p_{ij} is small, indicating x_{j} is dissimilar to x_{i}. Also, as ∑_{i} ∑_{j} p_{ij} = 1, i.e., p_{ij} is normalized and can therefore be considered as a probability that any two points x_{j} and x_{i} are similar to each other and are therefore neighbors in the feature space.

As a similarity measurement between any two points x_{i} and x_{j}, the conditional probability p_{j/i} defined above needs to be modified for two reasons.

First, the Gaussian kernel p_{ij} should be independent of the local density of the data points, which may vary drastically in different local regions of the feature space, so that similarities of different scales can be equally well represented. To achieve this goal, we adjust the variance σ_{i}^{2} in the Gaussian kernel p_{j/i} for each data point x_{i} in such a way that the entropy H(x_{i}) = − ∑_{j} p_{j/i} log_{2} p_{j/i} for all data points are the same as some prespecified value. As shown in Section B.2.1, the entropy of the Gaussian distribution N (x, μ, σ) is log(2πeσ^{2})/2, a monotonic function of the variance σ^{2}. The similarity measurement based on p_{j/i} of the same entropy for all data points is essentially the same as the Mahalanobis distance (Eq. (9.12) and the Bhattacharyya distance (Eq. (9.16)), in the sense that these similarity measurements are all normalized by the variance σ^{2}, and they are therefore independent with respect to the local data density and scale.

Second, as p_{j/i} ≠ p_{i/j} as a conditional probability defined above is asymmetrical, it needs to be modified so that it becomes symmetric as a similarity measurement should be. We therefore redefine the probability measurement as the average of two conditional probabilities:

p_{ij} = \frac{p_{i/j} + p_{j/i}}{2}  (11.98)

Once the dataset X in the high dimensional space is mapped to Y = [y_{1}, ⋯, y_{N}] in the low dimensional space, we also need to measure the similarity between any two data points. To do so, we further define such a similarity based on Student’s t-distribution of degree 1 (Eq. (B.140)):

q_{ij} = \frac{(1 + ||y_{i} − y_{j}||^{2})^{−1}}{∑_{k=1}^{N} ∑_{l=1}^{N}(1 + ||y_{k} − y_{l}||^{2})^{−1}} = \frac{(1 + d_{ij}^{2})^{−1}}{∑_{k=1}^{N} ∑_{l=1}^{N}(1 + d_{ij}^{2})^{−1}} = \frac{(1 + d_{ij}^{2})^{−1}}{D}

(11.99)

where D denotes the denominator:

D = ∑_{k=1}^{N} ∑_{l=1}^{N}(1 + d_{kl}^{2})^{−1}  (11.100)

and d_{kl} = ||y_{k} − y_{l}|| = [(y_{k} − y_{l})^{T} (y_{k} − y_{l})]^{1/2} is the Euclidean distance between any two data points y_{k} and y_{l}. Again as q_{ij} is normalized, it can be interpreted as a probability measurement of the similarity between any two points in the low dimensional space.

The t-distribution with one degree of freedom is also a bell shaped function, similar to the Gaussian distribution, but it has a lower peak in the center but higher tails on the sides, as shown in Fig. 11.11, which also illustrates why the similarity is measure based on the Gaussian distribution in the high-D space but

Figure 11.11 The t-distribution compared to the Gaussian distribution

the t-distribution in the low-D space. Specifically, consider any point x_{i} and two of its neighbors x_{j} and x′_{j} in the high-D space also illustrated in the figure:

• Point x_{j} = −0.9 (red on the left) similar to x_{i} = 0 as measured by the Gaussian in the high-D space is mapped to y_{j} = −0.6 in the low-D space measured by the t-distribution, even closer to y_{i} = x_{i}.

• Point x′_{j} = 1.3 (red on the right) not very similar to x_{i} = 0 as measured by the Gaussian in the high-D space is mapped to y′_{j} = 1.9 in the low-D space measured by the t-distribution, even farther away from y_{i} = x_{i}.

In other words, when all data points x_{n}, (n = 1, ⋯, N) in the high-D space are nonlinearly mapped to y_{n}, (n = 1, ⋯, N) in the low-D space in such a way that their similarities measured by the Gaussian before the mapping match those measured by the t-distribution after the mapping, similar points become even closer while dissimilar points become farther apart from each other. We note that whether two points are considered to be similar or not depends on the variance of the Gaussian, a hyperparameter that needs to be properly determined based on the specific dataset in question.

The goal of the tSNE method is to find the optimal mapping from all points in dataset X in the high-D space to Y in the low-D space that minimizes the discrepancy between the two sets of similarity measurements p_{ij} before the mapping and q_{ij} after the mapping, which can be quantitatively measured by the Kullback-Leibler divergence (Section B.2.2):

J = D_{KL}(P ||Q) = ∑_{i=1}^{N} ∑_{j=1}^{N} p_{ij} log \frac{p_{ij}}{q_{ij}} = ∑_{i=1}^{N} ∑_{j=1}^{N} (p_{ij} log p_{ij} − p_{ij} log q_{ij})

(11.101)

The KL-divergence is not symmetric as D_{KL}(P ||Q) ≠ D_{KL}(Q||P), i.e., it can not be viewed as a distance between the two sets of probabilities. However, this asymmetry is actually a desired feature. Consider the following two cases:

• If p_{ij} is large but q_{ij} is small, then D_{KL} is large, and such a mapping is heavily penalized. As the result, similar points that are close to each other in the high-D space tend to be mapped to points also close to each other in the low-D space.

• If p_{ij} is small but q_{ij} is large, then D_{KL} is small, and such a mapping is not heavily penalized. As the result, dissimilar points that are far apart in the high-D space are allowed to be mapped to points closer to each other in the low-D space.

We therefore see that minimizing the KL-divergence has the effect of preserving the local structure of the dataset, but not necessarily the global structure.

The optimal points in Y in the low-D space that minimize the objective function J = D_{KL}(P ||Q) can be found by the gradient descent method. The ith component ∂J/∂y_{i} of the gradient can be found by the chain rule, based on d_{ij} = d_{ji} treated as an intermediate variable:

\frac{∂J}{∂y_{i}} = ∑_{j=1}^{N} (\frac{∂J}{∂d_{ij}} \frac{∂d_{ij}}{∂y_{i}} + \frac{∂J}{∂d_{ij}} \frac{∂d_{ij}}{∂y_{i}} ) = 2 ∑_{j=1}^{N} \frac{∂J}{∂d_{ij}} \frac{∂d_{ij}}{∂y_{i}}  (11.102)

Note that J = D_{KL}(P ||Q) is a function of both p_{ij} as a constant, and q_{ij} = (1 + d_{ij}^{−1})/D, which in turn is a function of d_{ij}. The first derivative is:

\frac{∂J}{∂d_{ij}} = \frac{∂}{∂d_{ij}} [∑_{k=1}^{N} ∑_{l=1}^{N} (p_{kl} log p_{kl} − p_{kl} log q_{kl})]

= −\frac{∂}{∂d_{ij}} ∑_{k=1}^{N} ∑_{l=1}^{N} p_{kl} log q_{kl} = −\frac{∂}{∂d_{ij}} ∑_{k=1}^{N} ∑_{l=1}^{N} p_{kl}(log(q_{kl}D) − log D)

= − ∑_{k=1}^{N} ∑_{l=1}^{N} p_{kl} \frac{∂}{∂d_{ij}} log(q_{kl}D) + ∑_{k=1}^{N} ∑_{l=1}^{N} p_{kl} \frac{1}{D} \frac{∂D}{∂d_{ij}}  (11.103)

To proceed, we consider the two terms separately. There is only one non-zero term in the summation of the first term:

p_{ij} \frac{1}{q_{ij}D} \frac{∂}{∂d_{ij}} (1 + d_{ij}^{2})^{−1} = \frac{p_{ij}}{q_{ij}} \frac{2d_{ij}(1 + d_{ij}^{2})^{−2}}{D}  (11.104)

As ∑_{k=1}^{N} ∑_{l=1}^{N} p_{kl} = 1, the second term becomes:

\frac{1}{D} \frac{∂D}{∂d_{ij}} = \frac{1}{D} \frac{∂}{∂d_{ij}} (∑_{k=1}^{N} ∑_{l=1}^{N}(1 + d_{ij}^{2})^{−1}) = \frac{2d_{ij}(1 + d_{ij}^{2})^{−2}}{D}  (11.105)

Now Eq. (11.103) can be written as:

\frac{∂J}{∂d_{ij}} = 2d_{ij} (\frac{p_{ij}}{q_{ij}} − 1) \frac{(1 + d_{ij}^{2})^{−2}}{D} = 2d_{ij}(p_{ij} − q_{ij})(1 + d_{ij}^{2})^{−1}  (11.106)

The last equality is due to Eq. (11.99).

The second derivative in Eq. (11.102) is:

\frac{∂d_{ij}}{∂y_{i}} = \frac{∂}{∂y_{i}} [(y_{i} − y_{j})^{T} (y_{i} − y_{j})]^{1/2}
= \frac{1}{2} [(y_{i} − y_{j})^{T} (y_{i} − y_{j})]^{−1/2} 2(y_{i} − y_{j}) = \frac{y_{i} − y_{j}}{d_{ij}} (11.107)

Finally, substituting Eqs. (11.106) and (11.107) into Eq. (11.102) we get the gradient of the objective function J with respect to each of the map points y_{i}:

\frac{∂J}{∂y_{i}} = 2 ∑_{j=1}^{N} \frac{∂J}{∂d_{ij}} \frac{∂d_{ij}}{∂y_{i}} = 4 ∑_{j=1}^{N}(p_{ij} − q_{ij})(1 + d_{ij}^{2})^{−1}(y_{i} − y_{j})

= 4 ∑_{j=1}^{N} \frac{p_{ij} − q_{ij}}{1 + ||y_{i} − y_{j}||^{2}} (y_{i} − y_{j}) = 4 ∑_{j=1}^{N} w_{ij}(y_{i} − y_{j})

= (∑_{j=1}^{N} w_{ij}) y_{i} − ∑_{j=1}^{N} w_{ij}y_{j}  (11.108)

where w_{ij} = (p_{ij} −q_{ij})/(1+d_{ij}^{2}) for all i, j = 1, ⋯, N. Having found the gradient vector, we can iteratively approach the optimal data points in the low-D space that minimize the discrepancy J = D_{KL}(P ||Q):

y_{n+1} = y_{n} − δ \frac{∂J}{δy} + α_{n}(y_{n} − y_{n−1})  (11.109)

where δ is the step size (learning rate) and α is the coefficient for the momentum term in the direction of the previous iteration step (Eq. (2.56)).

The Metlab function below takes as input the data set X in the original high dimensional space, and generates as output the N × N matrix P of which the component p_{ij} in the ith row and jth column is defined in Eq. (11.98):

function P=data2prob(X)        % Given dataset X, find P matrix
    x2=sum(X.^2);              % squared distances between points
    D=x2+x2'-2*(X'*X);         % pair-wise squared distances
    N=size(D,1);               % number of data points
    P=zeros(N);                % initializing matrix P
    H0=3;                      % desired entropy value
    tol=1e-4;                  % tolerance
    for i=1:N                  % for each of the N data points
        d2=D(i,[1:i-1,i+1:N]); % squared distance between xi and others
        sigma=1;               % default value of variance
        sigmamax=9*sigma;      % initial upper limit
        sigmamin=0;            % initial lower limit
        H=0;
        while abs(H-H0)>tol    % adjust sigma for desired entropy
            p=exp(-d2/sigma);  % Gaussian kernel

    p=p/sum(p);        % normalization
    H=-sum(p.*log(p)); % actual entropy based on current sigma
    if H<H0            % H too small
       sigmaamin=sigma; % update upper limit
       sigma=(sigma+sigmamax)/2; % increase sigma
    else               % H too small
       sigmamax=sigma; % update lower limit
       sigma=(sigma+sigmamin)/2; % reduce sigma
    end
end
P(i,[1:i-1,i+1:N])=p;  % Save resulting p for xi
end
P=(P+P')/2;                % make P symmetric
P=P/sum(P(:));             % normalization
end

The Metlab function below takes as inputs matrix P generated by the previous function, dim as the dimension of the low dimensional space, and δ is the step size, and generates as output Y for the data points in the low dimensional space:

function Y=tSNE(P,dim,delta)
N=size(P,1);
Y=0.0001*randn(dim,N);   % random initialization of data
dY=zeros(dim,N);         % increment for gradient descent
alpha=0.5;               % coefficient for the momentum term
it=0;
while it<200
it=it+1;
Y0=Y;
s=sum(Y.^2);         % squared distances between points
T=1./(1+s+s'-2*(Y'*Y)); % t-distribution
Q=T./sum(T(:));      % normalization
W=(P-Q).*T;
g=Y*(diag(sum(W,1))-W); % gradient of KL-divergence
Y=Y-delta*g+alpha*dY % gradient descent with monumtum
Y=Y-mean(Y,2);       % remove mean from dataset
dY=Y-Y0;             % increment as momentum for next step
end
end

The tSNE method is applied to the dataset of handwritten digits of 2240 256-D data points to visualize the dataset in 2-D and 3-D spaces as shown in Fig. 11.12. Comparing these results with those shown previously in Fig. 10.16, we see that the tSNE method is obviously more effective in terms of separating the ten classes in 2 or 3 out of 256 dimensional space than the PCA method.

Figure 11.12 Visualization of 256-D dataset by the PCA (left) and tSNE (right) methods

Problems

1. Carry out kernel PCA to visualize the 4-D iris dataset in both 2-D and 3-D space, with all data points color coded according to their class identities. Compare your results with those obtained in the previous homework.

2. Carry out kernel PCA to visualize the 256-D handwritten digit dataset in both 2-D and 3-D space, with all data points color coded according to their class identities. Compare your results with those obtained in the previous homework.

3. Carry out probabilistic PCA to visualize the 256-D handwritten digit dataset in both 2-D and 3-D space, with all data points color coded according to their class identities. Compare your results with those obtained in the previous problem.

4. Use the PCoA method to reconstruct the locations of some cities in the US based on their pairwise distances, which can be obtained from their longitudelatitude coordinates available here: https://www.latlong.net/category/cities-236-15.html or here: https://www.geodatos.net/en/coordinates/united-states. Compare your reconstructed map with the ground truth.

5. Apply PCoA method to visualize the iris dataset in both 2-D and 3-D spaces and compare the results with those obtained by the PCA method.

6. Apply PCoA method to visualize the handwritten digit dataset in both 2-D and 3-D spaces and compare the results with those obtained by the PCA method.

7. Apply the tSNE method to visualize the handwritten digit dataset in both 2-D and 3-D spaces and compare the results with those obtained by the PCA method.

12 Independent Component Analysis

The Independent component analysis (ICA) is a computational method widely used in signal processing and data mining to reveal the independent signal components hidden in an observed dataset. The methods of ICA and PCA considered in the previous chapters are similar in the sense that they both intend to find some essential signal components embedded in the data. However, the ICA is different from the PCA as it is based on a different assumption that each of the variables in the observed multivariate data is some linear combinations of a set of independent signal component. These independent components are to be separated based on a fundamental statistic property that the distribution of a linear combination of a set of random variables tends to be more Gaussian than those of the individual variables.

A typical problem that can be addressed by the ICA is the blind source separation (BSS) problem, for the purpose of separating and reconstructing a set of independent source signals from their linear mixtures. For example, in the “cocktail party problem”, the goal is to reconstruct the individual voices of multiple speakers from a set of microphones recording all the conversations. The word “blind” means that no prior knowledge about either the sources or the mixing process is assumed to be available, except that the source signals are statistically independent. Although this BSS problem seems severely under constrained, the method of ICA can find nearly unique solutions satisfying certain properties.

In the following, we will first discuss the theoretical background of the ICA methods and then specifically consider two different ICA algorithms.

12.1 Independence and Non-Gaussianity

The goal of independent component analysis is to reconstruct a set of m independent source signals, called independent components and denoted by s = [s_{1}, ⋯, s_{m}]^{T}, from another set of n observed variables denoted by x = [x_{1}, ⋯, x_{n}]^{T}, each assumed to be a linear combination of the source signals:

x = As = ∑_{i=1}^{m} a_{i}s_{i}  (12.1)

where A = [a_{1}, ⋯, a_{m}] is an n × m mixing matrix of unknown values. We assume n ≥ m to avoid under determination. Given a dataset X = [x_{1}, ⋯, x_{N}] containing N observed data samples x_{n} = As_{n} (n = 1, ⋯, N), the ICA is to recover the independent source signals s_{1}, ⋯, s_{N}, based on a linear reconstruction model

y = Wx = WAs  (12.2)

where W is a m × n reconstruction matrix, and y = [y_{1}, ⋯, y_{m}]^{T} contains m reconstructed signals y_{i} = w_{i}^{T} x to match the corresponding independent signals s_{i} in s, and w_{i}^{T} is the ith row vector of W. It is obvious that for the reconstructed signal y to match exactly the source s, ideally we desire to find W = A^{−1}. But as A is unknown, we need to find W in some different way, as the main task of ICA algorithms.

As ICA is an unsupervised method without labeled training data for the ground truth regarding either A or s, we cannot find the reconstruction matrix W in closed form by the least squares method that minimizes the squared error ||y − s||^{2}. Instead we have to find the optimal W as well as s based only on the requirement that the components in y = Wx need to be as independent of each other as possible.

The method of ICA can be compared with the PCA. Both methods are to find a matrix W so that a given data vector x is mapped to another vector y = Wx. In PCA, the transform matrix W = V^{T} (Eq. (10.20)) is the eigenvector matrix of the covariance matrix Σ_{x} of x, i.e., the data point x is projected onto the eigenvectors v_{i} as the orthogonal basis of the space, and components y_{i} = v_{i}^{T} x of y are completely decorrelated. (Only in the special case when y = [y_{1}, ⋯, y_{n}] is Gaussian, are its components also independent.) While in ICA, W is in general not orthogonal, and the components are independent and necessarily also uncorrelated. In this sense, the ICA is a more powerful method as it satisfies a stronger requirement for W.

Here are some additional assumptions about the source signal s and the observed data x = As:

• The number n for the observed variables in x is no fewer than the number m for the independent source signals in s. We assume m = n in the following.

• The source signal s has a zero mean E[s] = 0 without loss of generality, and therefore so is the observed data: E[x] = E[As] = A E[s] = 0.

• The source components are stochastically independent and therefore uncorrelated, and they have to be non-Gaussian for reasons to be discussed below.

• The estimation of A and s is up to a scaling factor. Let

C = diag(c_{1}, ⋯, c_{n}), C^{−1} = diag(1/c_{1}, ⋯, 1/c_{n})  (12.3)

and A′ = AC^{−1} and s′ = Cs, then we have

x = As = [AC^{−1}][Cs] = A′s′  (12.4)

• Due to the arbitrary scaling factor, all independent components in s can be assumed to have unit variance σ_{i}^{2} = E[s_{i}^{2}] = 1. Also, as they are independent and necessarily uncorrelated, i.e.,

σ_{ij}^{2} = E[s_{i}s_{j}] = δ_{ij}, i.e. E(ss^{T}) = I  (12.5)

• The estimated independent components are not in any particular order. When the order of the corresponding elements in both s and A is rearranged, x = As still holds.

The theoretical foundation of ICA algorithms is the central limit theorem, which states that the distribution of the linear combination of N independent random variables approaches Gaussian as N → ∞. Specifically in a BSS problem, the observed variables in x = As as a linear combination of the independent components {s_{1}, ⋯, s_{m}} in s are necessarily more Gaussian than s. To reconstruct s from x, we need to reverse the process to reduce the Gaussianity. Based on this idea that non-Gaussianity is independence, the ICA method can be seen as the reverse process of the central limit theorem, with the goal of finding an optimal matrix W so that the components of the reconstructed y = Wx are least Gaussian and therefore most independent.

The ICA can therefore be considered as an optimization problem for finding W that maximizes certain objective function that measures either the non-Gaussianity or the independence of y = Wx. We consider two main ICA algorithms below. The first one, called FastICA, maximizes the non-Gaussianity of y, while the second one as a MLE method estimates matrix A by maximizing its likelihood, or equivalently the independence of the source signal s or the reconstructed signal y. Due to their different criteria, these two algorithms may perform differently.

12.2 Preprocessing and Whitening

Before discussing the specific algorithms, we first consider the following operations as the preprocessing of the observed data so that all assumptions about the data x made above and required by the algorithm are satisfied.

• Centering

Subtract E[x] from x so that it becomes centered with zero mean. By doing so, we also have E[x] = E[As] = A E[s] = 0, i.e., E[s] = A^{−1}E[x] = 0.

• Whitening

Find the eigenvalue matrix Λ and eigenvector matrix V of the covariance matrix E[xx^{T}] so that

V^{T} E[xx^{T}]V = Λ,  i.e., Λ^{−1/2}V^{T} E[xx^{T}]VΛ^{−1/2} = I (12.6)

As E[xx^{T}] is symmetric, V^{T} = V^{−1} is orthogonal. Then carry out a linear transform

x′ = (Λ^{−1/2}V^{T})x,  i.e., x = (Λ^{−1/2}V^{T})^{−1}x′ = (VΛ^{1/2})x′ (12.7)

so that the components of the data are not only decorrelated (same as the KLT transform) but also normalized with unity variance:

E[x′x′^{T}] = E[(Λ^{−1/2}V^{T} x)(Λ^{−1/2}V^{T} x)^{T}] = E[Λ^{−1/2}V^{T} xx^{T} VΛ^{−1/2}]
= Λ^{−1/2}V^{T} E[xx^{T}]VΛ^{−1/2} = I  (12.8)

i.e., x′ is whitened with covariance σ_{ij}^{2} = δ_{ij}.

Once x is centered and whitened to become x′ = (Λ^{−1/2}V^{T})x, the reconstruction becomes

y = Wx = W(VΛ^{1/2})x′ = W′x′  (12.9)

where W′ = WVΛ^{1/2} is the reconstruction matrix for whitened x′. We also have

E[x′x^{′T}] = E[(As)(As)^{T}] = A E[ss^{T}]A^{T} = AA^{T} = I  (12.10)

As y = W′x′ is to match s, we assume they have the same covariance E[yy^{T}] = E[ss^{T}] = I (Eq. (12.5)), and further get

I = E[yy^{T}] = E[(W′x′) (W′x′)^{T}] = W′ E[x′x^{′T}]W^{′T} = W′W^{′T}  (12.11)

We see that after centering and whitening, both the mixing matrix A and the reconstruction matrix W′ becomes orthogonal and can therefore be interpreted as rotation matrices. For convenience, in the following discussion we will still denote the whitened data samples by x and the reconstruction matrix by W.

12.3 Non-Gaussianity and FastICA

In ICA, we desire to find the optimal W that maximizes the non-Gaussianity of the reconstructed y = Wx. In general, the non-Gaussianity of a random variable y with a density function p(y) can be measured by

• Kurtosis

The excess kurtosis (Section B.2.4) of the centered and whitened random variable x (with μ_{x} = 0 and σ_{x} = 1) is defined as:

Kurt(x) = E{x^{4}} − 3  (12.12)

which is zero only when x has a Gaussian distribution. A non-Gaussian distribution is either supergaussian if its kurtosis is positive or subgaussian if its kurtosis is negative. Therefore the absolute value of the excess kurtosis or kurtosis squared can be used to measure the non-Gaussianity of a distribution. However, as kurtosis is very sensitive to noisy outliers, it is not a robust measurement of non-Gaussianity.

• Negentropy

The negentropy of a random variable y is defined as the difference between its entropy H(y) (Section B.2.1) and the entropy of a Gaussian variable g with the same variance σ^{2}:

J(y) = H(g) − H(y) = C − H(y) ≥ 0  (12.13)

where C = H(g) = log(2πeσ^{2})/2 ((Eq. (B.183)) is a constant independent of y. Here J(y) is non-negative as Gaussian distribution has the maximum entropy among all distributions with the same variance. Negentropy J(y) measures non-Gaussianity as it takes a larger value when y is less Gaussian. The negentropy defined for a scalar random variable y can be generalized to a random vector y = [y_{1}, ⋯, y_{m}]^{T}:

J(y) = H(y_{G}) − H(y) = C − H(y) ≥ 0  (12.14)

While kurtosis and negentropy of a random variable y are good measurement for the non-Gaussianity of y, they both depend on its distribution p(y), which is typically not readily available, and the negentropy J(y) as theoretically defined above can only be approximated as below based on the maximum-entropy principle:

J(y) ≈ ∑_{i} c_{i}(E[G_{i}(y)] − E[G_{i}(g)])^{2}  (12.15)

where g is again a Gaussian variable and both y and g are assumed to have zero mean and unit variance, and G_{i} is some non-quadratic function such as the following also shown in in Fig. 12.1:

G_{1}(y) = \frac{1}{a} log cosh (a y), G_{2}(y) = − exp(−y^{2}/2)  (12.16)

where 1 ≤ a ≤ 2 is a constant. This approximated negentropy is always greater than zero except when x is Gaussian. In particular, when only a single term is used in Eq. (12.15), we have

J(y) ≈ (E[G(y)] − E[G(g)])^{2}  (12.17)

Since the second term is a constant, we want to maximize E{G(y)} to maximize J(y).

Based on this approximated negentropy, the FastICA algorithm finds the reconstruction matrix W that maximizes the sum of the approximated negentropies of all m components of the reconstructed signal y = Wx:

∑_{i=1}^{m} E[G(y_{i})] = ∑_{i=1}^{m} E[G(w_{i}^{T} x)]  (12.18)

where y_{i} = w_{i}^{T} x is the ith component of y and w_{i}^{T} is the ith row vector of W.

We first consider computing w_{i} for y_{i} as one of the m components (with the subscript i dropped for now). Specifically, we need to find a vector w that

Figure 12.1 Two typical G-functions

maximizes E[G(y)] as the objective function subject to the constraint that w has to be normalized, i.e., w^{T} w = 1. To solve this constrained optimization problem we first construct the Lagrangian function:

L(w) = E[G(w^{T} x)] − \frac{λ}{2}(w^{T} w − 1)  (12.19)

and then set to zero its derivative with respect to w:

g_{L}(w) = \frac{d}{dw} L(w) = E[x \frac{d}{dw} G(w^{T} x)] − λw = E[x G′(w^{T} x)] − λw = 0 (12.20)

where G′(z) = dG(z)/dz is the derivative of function G(z). This equation system can be solved for w iteratively by the Newton-Raphson method:

w ⇐ w − J_{g}^{−1}(w)g(w)  (12.21)

Here J_{f}(w) is the Jacobian matrix of g(w) = dL(w)/dw:

J_{g}(w) = \frac{d}{dw} g(w) = E[xx^{T} G′′(w^{T} x)] − λI  (12.22)

where G′′(z) = d^{2}G(z)/d^{2}z is the second order derivative of G(z). We further approximate the first term on the right to get

E[xx^{T} G′′(w^{T} x)] ≈ E[xx^{T}] E[G′′(w^{T} x)] = E[G′′(w^{T} x)]I  (12.23)

where E[xx^{T}] = I as the dataset is assumed to be whitened, then the Jacobian becomes diagonal

J_{f}(w) = (E[G′′(w^{T} x)] − λ) I  (12.24)

and the Newton-Raphson iteration becomes:

w_{n+1} = w_{n} − \frac{1}{E[G′′(w_{n}^{T} x)] − λ} (E[x G′(w_{n}^{T} x)] − λw)  (12.25)

Multiplying both sides by c = λ − E[G′′(w_{n}^{T} x)], we get

cw_{n+1} = E[xG′(w_{n}^{T} x)] − E[G′′(w_{n}^{T} x)]w  (12.26)

where the scaling factor c can be removed as the updated w will be normalized.

This algorithm can then be repeated to get all row vectors in W each for one of the m row vectors w_{i}. However, if this process is carried out independently, the resulting vectors may not be orthogonal to each other as required. To address this issue, the Gram-Schmidt method in Section A.1.2 can be used. Specifically, during each iteration of the computation for w_{i} for i > 1, before it is normalized, we also subtract its projection onto each of the previously found w_{1}, ⋯, w_{i−1} (Eq. (A.39)), so that it is guaranteed to be orthogonal to all of them. All vectors such computed will guarantee that the final matrix W is orthogonal satisfying W^{T} = W^{−1} as required.

Now we have the following FastICA algorithm that is carried out m times each for one of the vectors w_{i}, (i = 1, ⋯, m):

1. Choose an initial random guess for w_{i}

2. Iterate:

w = E[xG′(w^{T} x)] − E[G′′(w^{T} x)]w  (12.27)

where the expectation is estimated as the average of all N samples in the dataset.

3.

w ⟵ w − ∑_{j=1}^{i−1}(w^{T} w_{j})w_{j}  (12.28)

4. Normalize:

w ⟵ w/||w||  (12.29)

5. If not converged, go back to step 2.

The Metlab code segment below shows the essential part of the FastICA algorithm, in which the reconstruction matrix W is calculated based on dataset X containing N observed data samples. HereImax is the maximum number of iterations and tol is the tolerance, e.g., 10^{−6}.

    syms u
    G=-exp(-u^2/2);
    G=log(cosh(u));             % symbolic function G
    g=matlabFunction(diff(G));  % G'
    dg=matlabFunction(diff(diff(G))); % G''
    [d N]=size(X);              % size of dataset
    [X T]=whiten(X);            % function that whitens dataset
    W=[];                       % initialize weight matrix W
    for i=1:d                   % for each weight vector w

w=rand(d,1);            % random initialization
for i=1:Imax
    wold=w;             % old weight vector
    wx=w'*X;
    w=X*g(wx)'-sum(dg(wx))*w; % new weight vector
    if i>1
        w=w-W*W'*w;     % Gram-Schmidt process
    end
    w=w/norm(w);        % renormalization
    if norm(w-wold)<tol % convergence
        break
    end
end
W=[W w];
end
Y=W'*X;                     % signal Reconstruction by ICA
W=W*T;

Also, listed below is the whitening function whiten used in the code above that takes the dataset X as input and generates the centered and whitened data and the whitening transform matrix T as the output:

function [X T]=whitening(X)
[d N]=size(X);
X=X-repmat(mean(X,2),1,N);
[U S V]=svd(X);
T=sqrt(N)*diag(1./(diag(S(1:d,1:d))))*U';
X=T*X;
end

12.4 Likelihood and Independence Maximization

The ideal reconstruction matrix W can also be estimated by solving an optimization problem based on the following different but equivalent criteria all related to the assumed independence of the source signal s, or, equivalently, the desired independence of the reconstructed signal y = Wx = WAs.

• The optimal W can be found so that the reconstructed signal y = Wx is maximally independent measured by the mutual information (Eq. (B.201)) shared by its components {y_{1}, ⋯, y_{m}}:

I(y) = ∑_{i=1}^{n} H(y_{i}) − H(y) = − ∑_{i=1}^{n} E [log p(y_{i})] − H(Wx)

= − E [∑_{i=1}^{n} log p(y_{i})] − log |det W| − H(x)  (12.30)

where H(y) = H(Wx) = H(x) + log | det W| (Eq. (B.188)), and the last term H(x) is unrelated to W and can therefore be dropped.

• Matrix A can be estimated so that p(x) of the observed data x most closely matches p(x|A) based on the assumed independence of the components in the source s (Eq. (B.85)):

p(x|A) = p(As) = | det A^{−1}| p(s) = | det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{i})  (12.31)

Now the optimal A can be found as the one that minimizes the KLdivergence between p(x and p(x|A):

D_{KL}(p(x)||p(x|A) = ∫ p(x) log \frac{p(x)}{p(x|A)}dx
= ∫ p(x) log p(x)dx − ∫ p(x) log (| det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{i})) dx

= H(x) − log | det A^{−1}| ∫ p(x)dx − ∫ p(x) (∑_{i=1}^{m} log p_{i}(s_{i})) dx

= H(x) − log | det A^{−1}| − E_{x} [∑_{i=1}^{m} log p_{i}(s_{i})]  (12.32)

Here the first term H(x) independent of A can be dropped, and the expectation with respect to the observed data x in the last term can be estimated as the average of the all samples in the dataset X = [x_{1}, ⋯, x_{N}].

• Matrix A can also be estimated by maximizing its likelihood given the observed dataset X containing N i.i.d. samples:

L(A|X) = p(X|A) = ∏_{n=1}^{N} p(x_{n}|A) = ∏_{n=1}^{N} | det A^{−1}| ∏_{i=1}^{m} p_{i}(s_{in})

= | det A^{−1}|^{N} ∏_{n=1}^{N} ∏_{i=1}^{m} p_{i}(s_{in})  (12.33)

where p(x_{n}|A) is given in Eq. (12.31) and s_{in} is the ith component of s_{n}.

We further get the log-likelihood:

l(A|X) = log L(A|X) = log [| det A^{−1}|^{N} ∏_{n=1}^{N} ∏_{i=1}^{m} p_{i}(s_{in})]

= N [log | det A^{−1}| + \frac{1}{N} ∑_{n=1}^{N} ∑_{i=1}^{m} p_{i}(s_{in})]

∝ log | det A^{−1}| + E_{x} [∑_{i=1}^{m} log p_{i}(s_{i})]  (12.34)

Here the average over all N data samples in the second term is treated as the estimated expectation with respect to x.

All three cases above are equivalent, as maximizing the log-likelihood in Eq. (12.34) is equivalent to minimizing mutual information of either the reconstructed signal y or the source signal s in Eq. (12.30) or (12.32), thereby maximizing their independence.

Also, we see that for the reconstructed signal y = Wx = WAs to match perfectly the source signal s, the reconstruction matrix simply needs to be W = A^{−1} so that y_{i} = w_{i}^{T} x = s_{i} for all i = 1, ⋯, m, and the three criteria above become exactly the same. We can now find the optimal W that maximizes Eq. (12.34) as the objective function:

J(W) = log | det W| + E [∑_{i=1}^{m} log p_{i}(w_{i}^{T} x)]  (12.35)

The gradient of this objective function is:

g_{J}(W) = \frac{d}{dW} J(W) = \frac{d}{dW} log | det W| + E [\frac{d}{dW} ∑_{i=1}^{m} log p_{i}(w_{i}^{T} x)] (12.36)

The first term is (Eq. (B.245)):

\frac{d}{dW} log | det W| = (W^{−1})^{T} = (W^{T})^{−1}  (12.37)

For each term in the summation of the second term, w_{i}^{T} x = y_{i} is the ith component of y, and the derivative with respect to the kth component w_{ki} of w_{i} is:

\frac{d}{d w_{ki}} log p_{i}(w_{i}^{T} x) = \frac{d}{d w_{ki}} log p_{i}(y_{i}) = \frac{p′_{i}(y_{i})}{p_{i}(y_{i})} x_{k} = ϕ_{i}x_{k}  (12.38)

where we have defined

ϕ_{i} = ϕ(y_{i}) = \frac{p′(y_{i})}{p(y_{i})}  (12.39)

If we further assume specifically p(y) = 1/ cosh(y) as a non-Gaussian distribution, and get

ϕ(y) = \frac{(1/ cosh(y))′}{1/ cosh(y)} = −\frac{sinh(y)}{cosh(y)} = − tanh(y)  (12.40)

then the second term of Eq. (12.36) can be written as E[ϕ(y)x^{T}], where ϕ(y) = [ϕ_{1}, ⋯, ϕ_{m}]^{T}, and the gradient becomes:

g_{J}(W) = (W^{−1})^{T} + E[ϕx^{T}]  (12.41)

Now the optimal W that maximizes J(W) can be found iteratively based on the gradient ascend method:

W_{n+1} = W_{n} + δg_{J}(W_{n}) = W_{n} + δ [(W_{n}^{−1})^{T} + E[ϕ(y)x^{T}]]  (12.42)

where δ is the step size, and the expectation can be estimated as the average of all N data samples.

The gradient g_{J}(W) can be modified by post-multiplying W^{T} W to get the natural gradient (Section B.3.4):

[(W^{T})^{−1} + E[ϕx^{T}]] W^{T} W = W + E[ϕ x^{T} W^{T} W] = W + E[ϕy^{T} W] (12.43)

The last equality is due to Wx = y. Now the iteration becomes:

W_{n+1} = W_{n} + δg_{J}(W_{n}) = W_{n} + δ (W_{n} + E[ϕy^{T} W_{n}])  (12.44)

Based this natural gradient along a modified search direction in each step of the iteration, it will converge more quickly.

The Metlab code segment listed below shows the essential part of the MLE algorithm, where Imax is the maximum number of iterations, \lambda is the step size, and tol is the tolerance, e.g., 10^{−6}. In the algorithm, data preprocessing (centering and whitening) is not required and is therefore optional. Here matrix W can be updated based on either the gradient or the natural gradient which converges much more quickly.

    [d N]=size(X);      % size of dataset
    T=eye(d);           % default whitening transform matrix
    if preprocessing    % optional preprocessing
        [X T]=whiten(X);
    end
    W=eye(d);           % initialize W
    lambda=0.5;         % step size
    tol=10^(-6);        % tolerance
    for i=1:Imax
        Y=W*X;          % reconstruction of y
%        Wnew=W+lambda*(inv(B)'-tanh(Y)*X'/N);
                                            % update W by gradient
        Wnew=W+lambda*(W-tanh(Y)*(Y'*W)/N); % or natural gradient
        if mean(abs(Wnew(:)-W(:))) < tol    % check convergence
            break
        end
        W=Wnew;

Figure 12.2 Reconstruction of independent components by ICA based on ML (2D)

    end
    W=W*T;

Example 12.1 This example demonstrates the ICA algorithm based on the maximum likelihood method for the reconstruction of two independent source signals in s = [s_{1}, s_{2}]^{T}, shown in the first panel in Fig. 12.2, based on the observed data x = [x_{1}, x_{2}]^{T} = A_{0}s, shown in the second panel in the figure.

The two reconstructed independent components by the ICA algorithm are shown in the third panel, which match the ground truth source signals in the first panel very well.

Also the estimated mixing matrix A is listed below, together with the ground truth A_{0} (not used in the algorithm):

A = [ \begin{matrix}0.243 & −0.111 \\ −0.120 & 0.218 \end{matrix} ] ,  A_{0} = [ \begin{matrix}2 & −1 \\ −1 & 2 \end{matrix} ]  (12.45)

We see that the ground truth A_{0} is matched by the estimated A very closely, when scaled up by a factor of 9.

In the second panel, we also plotted the column vectors of the ground truth mixing matrix A_{0} (black dashed lines), together with those of the estimated mixing matrix A (red lines). These two sets of lines match each other very closely, both representing the two independent sources.

For comparison, the PCA method is also used to find the principal components of the dataset, as shown by the blue lines in the fourth panel. They are also plotted in the second panel, as the eigenvectors of the covariance matrix of the dataset, weighted by the corresponding eigenvalues. As expected, these eigenvectors are along the major and minor axes of an ellipse approximating the overall distribution of observed data points. We see that the principal components are very different from the independent components..

The same algorithm is also applied to a 3-D dataset and the results are shown in Fig. 12.3.

Example 12.2 This example illustrates the results of the FastICA algorithm

Figure 12.3 Reconstruction of independent components by ICA based on ML (3D)

Figure 12.4 Reconstruction of independent components in time signals by FastICA

when applied to reconstruct the independent components of three observed time signals shown in the 2nd panel of Fig. 12.4, each as a linear combination of some three independent source signals (noisy square wave, triangular wave, and sinusoidal wave) as shown in the first panel.

The reconstructed independent components are shown in the third panel, which closely match the ground truth source signals (first panel), up to some scaling factors (could be negative). For comparison, the principal components obtained by the PCA method are also shown in the fourth panel, which are very different from the desired independent components.

Problems

1. Implement the ICA algorithm based on maximum likelihood method and

apply your code to both the 2D and 3D datasets used in Example 12.1. The independent source signals can be generated by the code below:

function S=data(d,N)
c=30;                    % variance of independent signals
if d==2                  % 2D signals
m=[0 0]';            % zero mean
C1=[1 0; 0 c];
C2=[c 0; 0 1];       % covariance of two independent sets
S=mvnrnd(m,C1,N)';   % generate 2 sets of N data points
S=[S mvnrnd(m,C2,N)'];  % with covariances C1 and C2
elseif d==3              % 3D signals
m=[0 0 0]';          % zero mean
C1=eye(3);  C2=C1;  C3=C1;
C1(1,1)=c;  C2(2,2)=c;  C3(3,3)=c;
                     % covariance of three independent sets
S=mvnrnd(m,C1,N)';   % generate 3 sets of N data points
S=[S mvnrnd(m,C2,N)'];  % with covariances C1, C2 and C3
S=[S mvnrnd(m,C3,N)'];
end
S = S./max(abs(S(:)));   % normailization
end

The observed data can then be obtained by linearly combining the independent source signals:

if d==2
A=[2 -1; -1 2];                % 2D mixing matrix
elseif d==3
A=[3 -2 1; -2 1 3; 1 2 -3];    % 3D mixing matrix
end
X=A*S;     % generate observed data by mixing independent sources

2. Implement the algorithm of FastICA and apply your code to some observed time signals each as a linear combination of some independent components (such as square, sinusoidal, and triangular waves of different frequencies) contaminated by some random noise, similar to those used in Example 12.4.

Part IV Classification

In Part IV, we consider pattern classification. As one of the most important topics in machine learning, classification can be considered as a learning process that finds out the underneath structure of the given dataset. Many real world problems can be formulated as to classify all data points in the given dataset into different categories. For example, in image recognition, the given data may be a set of images of various objects of interest (e.g., different animals or handwritten alphanumeric symbols), and the goal is to recognize these objects by classifying them into different categories each for one type of the objects. Such objects of interest are generally referred to as patterns, and the categories are the classes. In such cases, pattern classification is also called pattern (or object) recognition.

A classification problem can be generally formulated as to classify a set of N data samples, each represented as a vector x = [x_{1}, ⋯, x_{d}]^{T}, a point in the d-dimensional feature space, into a set of K classes denoted by {C_{1}, ⋯, C_{K}}, in such a way that all similar points are classified into the same class.

In the context of probabilistic methods of classification, these data points are assumed to be independent samples taken from a population with the same probability distribution, i.e., they can be treated as a set of independent and identically distributed (i.i.d.) samples of the population. Geometrically, the task of classifying similar data samples into the same classes can be considered as to partition the feature space into a set of regions each corresponding to one of the K classes. The boundary between any two such regions is called the decision boundary, and the goal of a specific classification algorithm, called a classifier, is to determine these decision boundaries based on the given dataset.

Classification is a supervised learning process if each sample x_{n} in the given dataset is associated with some labeling y_{n}, representing some prior knowledge of the data, such as the class identities of the samples. For example, we can use y_{n} ∈ {1, ⋯, N} to indicate sample x_{n} is known to belong to class C_{y_{n}}, or in the case of binary classification, we can use either y_{n} ∈ {0, 1} or y_{n} ∈ {−1, 1} to indicate x belongs to either of the two classes. How the samples are labeled depends on the specific classification algorithms.

Such a set of labeled data samples is called the training set and represented by D = {(x_{n}, y_{n}) (n = 1, ⋯, N)}, which can also be denoted by D = {X, y}, where X = [x_{1}, ⋯, x_{N}] is d × N array containing all N samples, and y = [y_{1}, ⋯, y_{N}]^{T} is an N-D vector containing their corresponding labelings. We can further denote by N_{k} the number of samples all labeled to belong to class C_{k}, so that N = ∑_{k=1}^{K} N_{k}.

Supervised classification is a two-stage process, in which a classifier is first constructed by learning the training set in the training stage, and then tested in the testing stage where any unlabeled data samples can be classified into one of the classes. We will consider different types of classification algorithms in Chapters 13 and 14.

If the samples in X are not labeled, i.e., y is unavailable in the given dataset D, then classification becomes an unsupervised learning process, called clustering, and the task becomes to group or cluster all data points into a set of clusters in such a way that similar data points (close to each other in the feature space) are in the same cluster while dissimular ones (far away from each other) are in different clusters. Unsupervised clustering is similar to supervised classification in the sense that they both partition the feature space into different regions each for one of the classes/clusters containing similar data samples, but clustering is very different from classification in the sense that the partitioning is totally based on how the data points are distributed in the feature space, without any prior knowledge, such as the labeling. We will consider various unsupervised clustering algorithms in Chapter 15.

Also in Chapter 15, we will consider a set of unsupervised algorithms for data clustering without the training set containing labeled data samples.

A wide variety of supervised and unsupervised learning algorithms will be discussed in the chapters of this Part, as well as those in Part V (for neural network methods), all for the same purpose of classification or clustering. It is natural to ask then, why do we need so many different algorithms? which of these algorithms is the best of them all? The answer is it depends. Every learning algorithm may be better suited for a certain type of learning problems than others, depending on many specifics such as the computational cost, the size of the dataset, how noisy or reliable the dataset is, how separable the classes are, etc. Moreover, even for the same learning algorithm, there are typically many hyperparameters to be adjusted in order to achive the optimal performance on a specific dataset. Therefore the reader is encouraged to experiment with different settings of the hyperparameters of each learning algorithm, and compare multiple algorithms for classification or clustering when applied to solve the same problem.

13 Statistic Classification

13.1 Discriminative vs. Generative Methods for Classification

Many supervised learning algorithms in both regression and classification are based on some assumed learning model with a set of parameters generally represented by a vector θ = [θ_{1}, ⋯, θ_{m}]^{T}. Once a specific model to use is determined, the main learning task is to obtain θ based on available training dataset.

All such learning methods can be characterized based on either of the following two types of models:

• Discriminative models

A discriminative algorithm is typically a deterministic method aiming at establishing a model that fits the training set in some optimal way, such as a set of decision boundaries that partitions the feature space into different regions each including all data points with the same labeling. In this case, only those data samples close to the boundaries play an important role, while all other samples farther away from the boundaries may be mostly ignored.

Here are some properties of the Discriminative methoda:

– The discriminative methods find the decision boundary in the feature space directly based on the data points in the training set, in general they ∘ are simpler than the generative approach requiring problem-specific

knowledge for building models of the data, ∘ are effective in producing accurate result when the dataset is large ∘ provide no insight or interpretation regarding the data and no un-

certainty estimate Typical discriminative methods include: ∘ K-nearest neighbors method ∘ Logistic regressions ∘ Support vector machines ∘ Perceptrons and Neural networks ∘ Decision trees and random forests

– Generative models The generative methods formulate the learning problem in a probabilistic framework to establish a model for the underlying structure

Figure 13.1 Discriminative vs. Generative Models

of the data as an effort to explain how the data was generated, and then finds specifically the decision boundaries based the model. Typically such a method finds certain probabilities of interest, such as the joint probability p(x, y|θ) of both x and the corresponding y in the training set, or the conditional probability p(y|x, θ) of the class symbol y, given the observed pattern x. In either case, the probability is parameterized by θ, to be estimated in the learning process based on the training set. Once θ is obtained, the probabilistic model becomes available, based on which the class identity of any unlabeled sample can be determined. For example, once the conditional probability p(y|x) is available, we can classify a data point x to class labeled by y with the greatest p(y|x), and we can further get the confidence for such a decision measured by the probability. For a probabilistic method, all data samples in the training set are needed for the estimation their overall probability densidy distribution.

Here are the properties of the generative methods:

∘ allow the use of problem-specific knowledge for building the model,

∘ can provide explanation and interpretation of the data,

∘ may be less prone to overfitting than a discriminative method,

∘ can provide uncertainty estimate

∘ may not be as accurate as the discriminative methods if the model does not fit the dataset well.

Typical generative methods include:

– Minimum distance classifier

– Naive Bayes classifier

– Gaussian mixture classifier

– Hidden Markov model

These two different models are illustrated in Fig. 13.1.

13.2 K Nearest Neighbors and Minimum Distance Classifiers

Here we first consider a set of simple supervised classification algorithms that assign an unlabeled sample x to one of the K known classes based on set N of training samples X = [x_{1}, ⋯, x_{N}], where each sample x_{n} is labeled by y_{n} = k ∈ {1, ⋯, K}, indicating it belongs to class C_{k}.

• k Nearest neighbors (k-NN) Classifier

Given an unlabeled pattern x, we first find its k nearest neighbors in the training dataset, and then assign x to one of the K classes by a majority vote of the k neighbors based on their class labelings. The voting can be weighted so that closer neighbors are more heavily weighted than those that are farther away. In particular, when k = 1, x is assigned to the class of its closest neighbor.

While the k-NN method is simple and straight forward, its computational cost is high as classifying any unlabeled pattern x requires computing distances to all data points in the training set.

• Minimum Distance Classifier

The minimum distance classifier is based on the most straight forward idea that an unlabeled data point x should be classified into the closest class in the feature space, among all K such classes {C_{1}, ⋯, C_{K}} in the training dataset:

if d(x, C_{k}) = min{d(x, C_{i}) i = 1, ⋯, C} then x ∈ C_{k}  (13.1)

where d(x, C_{k}) can be any of the point-to-class distances considered in Section 9.1, such as the centroid distance d(x, m_{k}) = ||x − m||, and the Mahalanobis distance:

d_{M}(x, C_{k}) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})  (13.2)

based on the estimated mean and covariance matrix of all data points belonging to the kth class C_{k}:

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N_{k}} x_{n},  Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N_{k}}(x_{n} − m)^{T} (x_{n} − m)  (13.3)

We need to compare different point-to-class distances to see which of them measures the similarity between x and each class C_{k} most accurately, as shown in the following examples.

Example 13.1 Here we consider a simple binary classification problem in a 1-D space as illustrated in the left panel in Fig. 13.2, based on the centroid distance, the Euclidean distance d_{2}(x, m_{k}) between the data point and the mean of all samples in each of the two classes represented by their

Figure 13.2 Binary Classification in 1-D

corresponding Gaussian pdfs:

C_{1} ∼ N (m_{1}, σ_{1}^{2}) = N (5, 1.2^{2}), C_{2} ∼ N (m_{2}, σ_{2}^{2}) = N (−5, 3^{2})

(13.4)

Using the centroid distance based on only the two means m_{1} = 5 and m_{2} = −5, a data sample x = 1 is classified to class C_{1} with a closer distance to its mean: d_{2}(x, m_{1}) = 4 < d_{2}(x, m_{2}) = 6. However, as shown in the figure, x seems to be more similar to class C_{2}, as it is more widely distribution than C_{1} due to its greater variance σ_{2} = 3 > σ_{1} = 1.2, which is not taken into consideration in this case.

Example 13.2 Here we use the Mahalanobis distances between x and each of the two classes, so that the variances of the two classes as well as their means are taken into consideration. In the 1-D case this distance is simply d_{M}(x, C_{k}) = (x−m_{k})^{2}/σ_{k}^{2}, which is proportional to (x−m_{k})^{2} but inversely proportional to σ_{k}^{2}. As d_{M}(x_{1}, m_{1}) = 11.11 > d_{M}(x_{1}, m_{2}) = 4.0, x = 1 is classified to C_{2}.

Example 13.3 Now consider classifying the two samples x_{1} = 1 and x_{2} = 3 shown in the right panel in Fig. 13.2 into either of two classes both with a normal distribution:

C_{1} ∼ N (0, 1.2^{2}),  C_{2} ∼ N (0, 3^{2})  (13.5)

The Mahalanobis distances d_{M}(x_{i}, C_{k}) = (x_{i} − m_{k})^{2}/σ_{k}^{2}, (i = 1, 2, k = 1, 2) between each of the two samples and each of the two classes are listed below:

class 1  class 2
x_{1} 1/1.2^{2} = 0.69 1/3^{2} = 0.11
x_{2} 3^{3}/1.2^{2} = 6.25 3^{2}/3^{2} = 1

(13.6)

As m_{1} = m_{2} = 0, |x_{i} − m_{1}|^{2} = |x_{i} − m_{2}|^{2} for both x_{1} and x_{2}, their Mahalanobis distances are determined by the variances σ_{2}^{2}, σ_{2}^{2} alone, and they are both classified into C_{2}, with a greater variance σ_{2}^{2} > σ_{2}^{2} and therefore smaller distance. However, as shown in the figure, it seems x_{1} = 1 should be classified to C_{1}. We therefore see that sometimes minimum distance classification based on the Mahalanobis distance may not be reliable, and some better method need to be considered, to be discussed in the following section.

13.3 Naive Bayes Classification

The method of naive Bayes (NB) classification is a classical supervised classification algorithm, which is first trained by a training set of N samples X = [x_{1}, ⋯, x_{N}] and their corresponding labelings y = [y_{1}, ⋯, y_{N}]^{T}, and then used to classify any unlabeled sample xT into class C_{k} with the maximum posterior probability. As indicated by the name, naive Bayes classification is based on Bayes’ theorem:

P (C_{k}|x) = \frac{p(x, C_{k})}{p(x)} = \frac{p(x|C_{k})P (C_{k})}{p(x)}  (13.7)

where

• P (C_{k}), or briefly P_{k}, is the prior probability that any randomly selected sample x belongs to class C_{k} without any prior knowledge of the sample. If the training set is a fair representation of all patterns of different classes in the entire population, in the sense that the data points in training set are uniformly sampled from the distribution they represent, then we can assume:

P_{k} = \frac{N_{k}}{N} = \frac{N_{k}}{∑_{l=1}^{K} N_{l}} , (k = 1, ⋯, K) satisfying ∑_{k=1}^{K} P_{k} = 1

(13.8)

where N_{k} is the number of data samples in the training set labeled to belong to class C_{k}. This estimation is based on the assumption that the training samples are evenly drawn from the entire population, and are therefore a fair representation of all K classes.

• p(x|C_{k}) = L(C_{k}|x) is the likelihood for any observed x to belong to C_{k}, which is proportional to the conditional probability of x given x ∈ C_{k}, assumed to be a Gaussian in terms of the mean vector m_{k} and covariance matrix Σ_{k}:

p(x|C_{k}) = N (x, m_{k}, Σ_{k}) = \frac{1}{(2π)^{d/2} |Σ_{k}|^{1/2}} exp [−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})]

(13.9)

This assumption is based on the fact that the Gaussian distribution has the maximum entropy (uncertainty) among all probability density functions with the same covariance, i.e., it imposes the least amount of unsupported constraint on the model for the dataset.

• p(x, C_{k}) is the joint probability of x and C_{k}:

p(x, C_{k}) = p(x|C_{k})P (C_{k}) = P (C_{k}|x)p(x)  (13.10)

• p(x) is the distribution of any data sample in the dataset, independent of its class identity, the weighted sum of all likelihood p(x|C_{k}) for k = 1, ⋯, K, i.e., the joint probability p(x, C_{k}) marginalized over all K classes:

p(x) = ∑_{k=1}^{K} p(x, C_{k}) = ∑_{k=1}^{K} P_{k} p(x|C_{k})

= ∑_{k=1}^{K} P_{k} [\frac{1}{(2π)^{d/2} |Σ_{k}|^{1/2}} exp (−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}))] (13.11)

• P (C_{k}|x) is the posterior probability that a data sample x belongs to class C_{k} based on the observed values of x.

Based on Bayes’ theorem given above, the naive Bayes method classifies an unlabeled data sample x to class C_{k} with the maximum posterior probability (maximum a posteriori (MAP)):

if P (C_{k}|x) = max_{l}{P (C_{l}|x), (l = 1, ⋯, K)}, then x ∈ C_{k}  (13.12)

As p(x) is common to all K classes (independent of k), it plays no role in the relative comparison among the K classes, and can therefore be dropped, i.e., x is classified to C_{k} with maximal p(x|C_{k}) P_{k}. In the case where the P_{k} is not available due to the lack of any prior knowledge, it has to be assumed to be the same for all K classes (equaly likely), then it is dropped and the method becomes maximum likelihood estimation (MLE).

The naive Bayes classifier is an optimal classifier in the sense that the classification error is minimum. To see this, consider an arbitrary boundary between two classes C_{1} and C_{2} that partitions the 1-D feature space into two regions R_{1} and R_{2}, as shown in Fig. 13.3. The probability of a misclassification is the joint probability P (x ∈ C_{i} ∩ x ∈ R_{j}) for a sample x ∈ C_{i} but falling in R_{j}. The total probability of error due to misclassification can be expressed as:

P (error) = P ((x ∈ R_{2}) ∩ (x ∈ C_{1})) + P ((x ∈ R_{1}) ∩ (x ∈ C_{2}))
= P (x ∈ R_{2}/C_{1}) P_{1} + P (x ∈ R_{1}/C_{2}) P_{2}
= P_{1} ∫_{R_{2}} p(x/C_{1})dx + P_{2} ∫_{R_{1}} p(x/C_{2})dx  (13.13)

As shown in Fig. 13.3, it is obvious that the Bayes classifier is indeed optimal, due to the fact that the decision boundary p(x|C_{1})P_{1} = p(x|C_{2})P_{2} between two classes C_{1} and C_{2} (the bottom plot) guarantees the classification error (shaded area) to be minimum.

To find the likelihood function p(x|C_{k}) = N (x, m_{k}, Σ_{k}), we need to first

Figure 13.3 Minimum Error of Naive Bayes Classifier

estimate m_{k} and Σ_{k} based on the N_{k} training samples in class C_{k} in the training set:

m_{k} = \frac{1}{N_{k}} ∑_{i=1}^{N_{k}} x_{i}, Σ_{k} = \frac{1}{N_{k}} ∑_{i=1}^{N_{k}}(x_{i} − m_{k})(x_{i} − m_{k})^{T} , (x_{i} ∈ C_{k}) (13.14)

Once both P_{k} = N_{k}/N and p(x|C_{k}) = N (x, m_{k}, Σ_{k}) are available, the classifier is trained, and the posterior P (C_{k}|x) can be calculated for the classification in Eq. (13.12).

As the classification is based on the relative comparison of the posterior P (C_{k}|x), any monotonic mapping of the posterior, such as the logarithmic function, can be equivalently used to simplify the computation:

log P (C_{k}|x) = log [p(x|C_{k})P_{k}/p(x)] = log p(x|C_{k}) + log P_{k} − log p(x)
= −\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k})
−\frac{N}{2} log(2π) − \frac{1}{2} log |Σ_{k}| + log P_{k} − log p(x)  (13.15)

We can further drop the constant terms log p(x) and N log(2π)/2 shared by all classes and get the quadratic discriminant function:

D_{k}(x) = −\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) − \frac{1}{2} log |Σ_{k}| + log P_{k}  (13.16)

Now the classification can be represented by

if D_{l}(x) = max{D_{k}(x), (k = 1, ⋯, K)}, then x ∈ C_{l}  (13.17)

Equivalently, we can also treat the negative of this discriminant function as a

Figure 13.4 Binary Classification in 1-D Revisited

distance between point x and class C_{k} with m_{k} and Σ_{k}:

d_{k}(x, C_{k}) = (x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) + log |Σ_{k}| − 2 log P_{k}  (13.18)

for minimum distance classification. We note that the first term is the Mohalanobis distance between x and C_{k}. However, the additional terms in the expression contribute to better classification performance, as illustrated in the example below.

Example 13.4 Reconsider Example 13.3 shown in the left panel of Fig. 13.4, but now using the negative of the discriminant function treated in Eq. (13.18) as the distance. In this case, the last term logP_{k} of the distance are dropped as they are assumed to be the same for both classes, and the distances d(x_{i}, C_{k}) = (x_{i} − m_{k})^{2}/σ_{k}^{2} − log |σ_{k}^{2}|, (i = 1, 2, k = 1, 2) between each of the two samples and each of the two classes are listed below:

class 1  class 2
x_{1} 0.69 + log(1.2^{2}) = 1.06 0.11 + log(3^{2}) = 2.31
x_{2} 6.25 + log(1.2^{2}) = 6.61 1 + log(3^{2}) = 3.20

(13.19)

As d(x_{1}, C_{1}) = 1.06 < d(x_{1}, C_{2}) = 2.31, x_{1} is classified into class C_{1} as desired, and as d(x_{1}, C_{1}) = 1.06 < d(x_{1}, C_{2}) = 2.31, x_{2} is still classified into C_{2}, the same as before. We see that the previous misclassification of x_{1} into C_{2} based on the Mahalanobis distance is now corrected, due to the additional term log(σ^{2}). Moreover, the right panel of Fig. 13.4 shows how the 1-D space is partitioned by the decision boundaries (the two vertical lines) into regions for C_{1} (inside the two vertical lines) and C_{2} (outside the vertical lines).

In the more general case of higher dimensional multiple classes, the feature space is partitioned into regions corresponding to the K classes by the decision boundaries between every pair of classes C_{i} and C_{j}, represented by the equation D_{i}(x) = D_{j}(x):

−\frac{1}{2}(x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i}) − \frac{1}{2} log |Σ_{i}| + log P_{i}

= −\frac{1}{2}(x − m_{j})^{T} Σ_{j}^{−1}(x − m_{j}) − \frac{1}{2} log |Σ_{j}| + log P_{j}  (13.20)

which can be written as

x^{T} Wx + w^{T} x + w = 0  (13.21)

where

W = −\frac{1}{2}(Σ_{i}^{−1} − Σ_{j}^{−1})
w = Σ_{i}^{−1}m_{i} − Σ_{j}^{−1}m_{j}
w = −\frac{1}{2}(m_{i}^{T} Σ_{i}^{−1}m_{i} − m_{j}^{T} Σ_{j}^{−1}m_{j}) − \frac{1}{2} log \frac{|Σ_{i}|}{|Σ_{j}|} + log \frac{P_{i}}{P_{j}} (13.22)

In general, this decision boundary is a quadratic hypersurface (hypersphere, hyperellipsoid, hyperparabola, or hyperhyperbola), by which an unlabeled data sample x is classified into either of the two classes C_{i} and C_{j} based on whether it is on the positive or negative side of the surface::

if x^{T} Wx + w^{T} x + w \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then x ∈ \{ \begin{matrix}C_{i} \\ C_{j} \end{matrix}  (13.23)

We further consider several special cases:

• All classes have the same prior:

P_{i} = P_{j} (i, j = 1, ⋯, K)  (13.24)

then the last term of D_{i}(x) is zero, and the naive Bayes classification as a MAP method becomes an MLE method, we have

D_{k}(x) = −\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}) − \frac{1}{2} log |Σ_{k}|  (13.25)

• All classes have the same covariance matrix Σ_{i} = Σ, the discriminant function becomes:

D_{i}(x) = −\frac{1}{2}(x − m_{i})^{T} Σ^{−1}(x − m_{i}) + log P_{i}  (13.26)

Moreover, if all P_{i} are the same and the second term is dropped, D_{i}(x) becomes the negative Mahalanobis distance.

The boundary equation D_{i}(x) = D_{j}(x) between C_{i} and C_{j} becomes

−\frac{1}{2}(x − m_{i})^{T} Σ^{−1}(x − m_{i}) + log P_{i} = −\frac{1}{2}(x − m_{j})^{T} Σ^{−1}(x − m_{j}) + log P_{j}

(13.27)

As the quadratic terms x^{T} Σ^{−1}x on both sides of the equation are the same, it becomes a linear equation:

w^{T} x + w = 0  (13.28)

where

w = Σ^{−1}(m_{i} − m_{j})  (13.29)

and

w = −\frac{1}{2}(m_{i}^{T} Σ^{−1}m_{i} − m_{j}^{T} Σ^{−1}m_{j}) + log \frac{P_{i}}{P_{j}}  (13.30)

This linear equation represents a hyperplane between the two points m_{i} and m_{j} and perpendicular to the straight line Σ^{−1}(m_{i} − m_{j}) (the straight line m_{i} − m_{j} rotated by matrix Σ^{−1}).

• All classes have the same isotropic distribution:

Σ_{i} = σ^{2} I = diag[σ^{2}, ⋯, σ^{2}]  (13.31)

then |Σ_{i}| = σ^{2d} and D_{i}(x) becomes

D_{i}(x) = −\frac{||x − m_{i}||^{2}}{2σ^{2}} + log P_{i}  (13.32)

Note that the term log |Σ_{i}| has been dropped from the original expression of D_{i}(x) as it is now the same for all classes.

The boundary equation D_{i}(x) = D_{j}(x) becomes:

\frac{||x − m_{i}||^{2}}{2σ^{2}} − log P_{i} = \frac{||x − m_{j}||^{2}}{2σ^{2}} − log P_{j}  (13.33)

which can be simplified to a linear equation:

w^{T} x + w = 0  (13.34)

where

w = m_{i} − m_{j}, w = −(m_{i}^{T} m_{i} − m_{j}^{T} m_{j}) + 2σ^{2} log \frac{P_{i}}{P_{j}}  (13.35)

This linear equation represents a hyperplane between the two points m_{i} and m_{j} and perpendicular to the straight line passing through these points.

• Further, if all classes have the same P_{i}, D_{i}(x) becomes

D_{i}(x) = −(x − m_{i})^{T} (x − m_{i}) = −||x − m_{i}||^{2}  (13.36)

and the Bayes classifier becomes minimum distance classifier based on the centroid distance, the Euclidean distance between x and the mean m_{k} of class C_{k} (maximizing D_{i}(x) is equivalent to minimizing ||x − m_{i}||^{2}).

The Matlab functions for training and testing are listed low:

function [M S P]=NBtraining(X,y)   % naive Bayes training
[d N]=size(X);                 % dimensions of dataset
K=length(unique(y));           % number of classes
M=zeros(d,K);                  % mean vectors
S=zeros(d,d,K);                % covariance matrices
P=zeros(1,K);                  % prior probabilities

    for k=1:K                      % for each of K classes
        idx=find(y==k);            % indices of samples
        Xk=X(:,idx);               % get all samples in class
        P(k)=length(idx)/N;        % prior probability
        M(:,k)=mean(Xk');          % mean vector
        S(:,:,k)=cov(Xk');         % covariance matrix
    end
end
function yhat=NBtesting(X,M,S,P)   % naive Bayes testing
    [d N]=size(X);                 % dimensions of dataset
    K=length(P);                   % number of classes
    for k=1:K
        InvS(:,:,k)=inv(S(:,:,k)); % inverse of covariance
        Det(k)=det(S(:,:,k));      % determinant of covariance
    end
    for n=1:N                      % for each of N samples
        x=X(:,n);
        dmax=-inf;
        for k=1:K
            xm=x-M(:,k);
            d=-(xm'*InvS(:,:,k)*xm)/2;
            d=d-log(Det(k))/2+log(P(k)); % discriminant function
            if d>dmax
                yhat(n)=k;         % assign nth sample to kth class
                dmax=d;
            end
        end
    end
end

The classification result in terms of the estimated class identity ŷ of the training samples compared with the ground truth class labeling y can be represented in the form of a K × K confusion matrix, of which the element in the ith row and jth column is the number of training samples labeled as a member of the ith class but actually classified into the jth class. For an ideal classifier that classifies all data samples correctly, this confusion matrix should be a diagonal matrix. But for a nonideal classifier, its error rate can be found as the percentage of the number of misclassified samples (sum of all off-diagonal elements) out of the total number of samples. The code below shows the function that generates such a confusion matrix based on the estimated class identity ŷ and the ground truth labeling y:

function [Cm er]=ConfusionMatrix(yhat,ytrain)
    N=length(ytrain);            % number of test samples

Figure 13.5 Naive Bayes Classification of Concentric Binary Classes

    K=length(unique(ytrain));    % number of classes
    Cm=zeros(K);                 % the confusion matrixxs
    for n=1:N
        i=ytrain(n);
        j=yhat(n);
        Cm(i,j)=Cm(i,j)+1;
    end
    er=1-sum(diag(Cm))/N;        % error percentage
    end

Example 13.5 This example shows the classification of two cocentric classes with one surrounding the other in 2-D space, as shown in Fig. 13.5. The feature space is partitioned into two regions by an ellipse and 9 out of 200 samples misclassified, i.e., with an error rate of 9/200 = 4.5%. The confusion matrix is shown below:

[ \begin{matrix}92 & 8 \\ 1 & 99 \end{matrix} ]

indicating 8 of the 100 samples belonging to class 1 are misclassified to class 2, and 1 of the 100 samples belonging to class 2 is misclassified to class 1.

Example 13.6 This example shows the classification of a 2-class exclusive-or (XOR) data set with significant overlap, as shown in Fig. 13.6, in terms of the confusion matrix and the error rate of 61/400 = 15.25%. Note that the decision boundaries are a pair of hyperbolas. The confusion matrix is

[ \begin{matrix}175 & 25 \\ 36 & 164 \end{matrix} ]

Example 13.7 The left panel of Fig. 13.7 shows data points in 2-D space drawn from three distributions shown in the middle panel, while the right one shows the partitioning of the space by quadratic decision boundaries obtained by the

Figure 13.6 Naive Bayes Classification of XOR Binary Classes

Figure 13.7 Naive Bayes Classification of Three Classes

naive Bayes classifier. The confusion matrix of the classification result is shown below, with the error rate 49/600 = 8.17%.

[ \begin{matrix}196 & 3 & 1 \\ 0 & 191 & 9 \\ 3 & 33 & 164 \end{matrix} ]

Example 13.8 Fig. 13.8 shows a 3-D feature space containing K = 3 groups of data points (in red, green and blue), which is partitioned by three quadratic decision boundaries D_{i}(x) = D_{j}(x) (Eq. (13.21)) into three regions for each C_{l} of the three classes with maximum discriminant function D_{l}(x) (Eq. (13.17).

The same classification result can also be reached by carrying out three binary classifications each based on one of the three decision boundaries followed by a majority vote. For example, a data point in the red group will win at least two out of the three binary classifications and it is therefore classified into the red class. This method can be used to carry out multiclass classification based on any binary classifier, such as the support vector machine, to be considered in Chapter 14.

Example 13.9 Fig. 13.9 shows a subset of the dataset of handwritten digits each in a 16 × 16 image form treated as a 256 dimensional vector.

Figure 13.8 Partitioning of a 3-D Space by Quadratic Decision Boundaries

The dataset is visualized by mapping the data points from the original 256-D space to 3-D or even 2-D space based on either the linear KLT transform (Section 10.2), or the nonlinear tSNE (Section 11.6), as shown respectively in the left and right panels of Figs. 13.10 and 13.11.

To use the naive-Bayes classifier for the classification of this dataset, we need to estimate the covariance matrix Σ_{k} based on the 224 samples in each of the K = 10 class. As the rank of the matrix is 224−1 = 223 < 256, it is not invertible and Σ_{k}^{−1} needed in the algorithm does not exist. We therefore have to reduce the dimensionality by, either KLT or tSNE, from 256 to some value, e.g., 100, smaller than the number of samples used for estimating the covariance matrix.

The classifier is first trained on half of the data samples randomly selected samples and then tested on the other half. The classification results are given below in terms of the confusion matrices of the taring samples (first) and the testing samples (second). All 1120 training samples are correctly classified with

13.3 Naive Bayes Classification 317

Figure 13.9 Samples of Handwritten Digits Figure 13.10 2-D Visualization based on PCA (left) and tSNE (right)

zero error rate, while out of the remaining 1120 test samples, 198 are misclassified with an error rate of 17.7%.

[ \begin{matrix}107 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 105 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 118 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 110 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 120 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 128 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 104 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 109 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 111 \end{matrix} ]

Figure 13.11 3-D Visualization based on PCA (left) and tSNE (right)

[ \begin{matrix}101 & 0 & 0 & 0 & 0 & 0 & 0 & 2 & 0 & 0 \\ 0 & 58 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 105 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 5 & 2 & 6 & 101 & 2 & 4 & 0 & 28 & 4 & 4 \\ 0 & 15 & 0 & 0 & 102 & 0 & 0 & 4 & 0 & 9 \\ 5 & 4 & 1 & 1 & 0 & 99 & 1 & 1 & 2 & 1 \\ 0 & 17 & 1 & 0 & 2 & 0 & 95 & 2 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 62 & 0 & 0 \\ 6 & 15 & 6 & 4 & 6 & 1 & 0 & 18 & 109 & 9 \\ 0 & 3 & 0 & 0 & 2 & 0 & 0 & 2 & 0 & 90 \end{matrix} ]

The dimensionality of the dataset can be reduced further down to some value much lower such as a single digits, by either the PCA or tSNE methods, and then the naive Bayes classification is carried out in the low dimensional space. The error rates of the two classification methods when applied to this dataset (with all 2240 samples used in both training and testing) are compared in the table below and plotted in Fig. 13.12. It is interesting to note that when the dimensionality is reduced to lower than 7, the error rates based on tSNE are much lower than those based on PCA, but for higher dimensionalities, PCA slightly outperforms tSNE.

Dimensions: 1  2  3  4  5  6  7  8  9  10
PCA:  0.69 0.48 0.25 0.18 0.15 0.11 0.08 0.06 0.06 0.05
tSNE:  0.50 0.15 0.10 0.08 0.08 0.08 0.08 0.08 0.08 0.07

Figure 13.12 Comparison of KLT and tSNE for Feature Selection

13.4 Adaptive Boosting

Adaptive boosting (AdaBoost) is a supervised classification algorithm that belongs to the general method of ensemble learning which is first discussed in Section 4.4. AdaBoost is a binary classifier trained based on the training set containing N labeled data samples {(x_{n}, y_{n}) | n = 1, ⋯, N}, where each sample x_{n} is labeled by y_{n} ∈ {−1, +1}, indicating to which of the two classes C_{−} and C_{+} it belongs.

AdaBoost learning is an iterative process. In the t-th iteration, each of the N training samples is classified into one of the two classes by a weak classifier h_{t}(x_{n}) ∈ {−1, +1}, as a hypothesis. Here the classifier is said to be weak in the sense that its performance only needs to be better than chance. If x_{n} is correctly classified, then h_{t}(x_{n}) = y_{n} = ±1, i.e., y_{n} h_{t}(x_{n}) = 1, otherwise it is misclassified with h_{t}(x_{n}) = −y_{n} = ±1, i.e., y_{n} h_{t}(x_{n}) = −1.

The weighted overall error rate is defined as:

ε_{t} = \frac{∑_{y_{n} h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}  (13.37)

where w_{t}(n) is the weight for the nth training sample x_{n} at the t-th iteration. For this error rate to be lower than chance, it is required that ε_{t} < 1/2. We need to find the best weak classifier that minimizes this error ε_{t}. We also find the correct rate as

1 − ε_{t} = 1 − \frac{∑_{y_{n} h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)} = \frac{∑_{y_{n} h_{t}(x_{n})=1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}  (13.38)

At the beginning of the iteration with t = 0, all N weights are initialized to 1, w_{0}(1) = ⋯ = w_{0}(N) = 1, and the weighted error

ε_{0} = \frac{1}{N} ∑_{y_{n} h_{0}(x_{n})=−1} w_{0}(n) = \frac{number of misclassified samples}{total number of samples}  (13.39)

is the probability for any training sample x to be misclassified by the weak classifier h_{0}. During the iteration when t > 0, this weight w_{t}(n) will be modified depending on whether x_{n} is classified correctly in the subsequent iterations.

At each iteration, we also construct a strong or boosted classifier H_{t}(x_{n}) as a linear combination (boosting) of all previous weak classifiers h_{i}(x_{n}), (i ≤ t):

F_{t}(x_{n}) = α_{t}h_{t}(x_{n}) + F_{t−1}(x_{n}) = α_{t}h_{t}(x_{n}) + α_{t−1}h_{t−1}(x_{n}) + F_{t−2}(x_{n})
= ⋯ = ∑_{i=1}^{t} α_{i}h_{i}(x_{n})  (n = 1, ⋯, N)  (13.40)

where α_{i} is some coefficient for h_{i}(x_{n}) in the i-th iteration, to be determined as shown below. The strong classifier is then defined as the sign function of F_{t}(x):

H_{t}(x_{n}) = sign[F_{t}(x_{n})] = \{ \begin{matrix}+1 & F_{t}(x_{n}) > 0 \\ −1 & F_{t}(x_{n}) < 0 \end{matrix}  (13.41)

by which a training sample x_{n} is classified into one of the two classes. The magnitude |F_{t}(x_{n})| represents the confidence of this decision.

The weight w_{t}(n) of x_{n} in the t-th iteration (t > 0) is defined as

w_{t}(n) = e^{−y_{n}F_{t−1}(x_{n})} \{ \begin{matrix}> 1 & if y_{n} F_{t−1}(x_{n}) < 0 (misclassification) \\ < 1 & if y_{n} F_{t−1}(x_{n}) > 0 (correct classification) \end{matrix}

(13.42)

which can be obtained from w_{t−1}(n) of the previous iteration:

w_{t}(n) = e^{−y_{n} F_{t−1}(x_{n})} = e^{−y_{n} [α_{t−1}h_{t−1}(x_{n})+F_{t−2}(x_{n})]}
= e^{−α_{t−1} y_{n}h_{t−1}(x_{n})} e^{−y_{n} F_{t−2}(x_{n})} = e^{−α_{t−1} y_{n}h_{t−1}(x_{n})} w_{t−1}(n)(13.43)

and this can be carried out recursively all the way back to the first iteration with w_{1}(k) = e^{−α_{0} y_{n}h_{0}(x_{n})}w_{0}(n).

The performance of the strong classifier can be measured by the exponential loss, defined as the sum of all N weights:

E_{t+1} = ∑_{n=1}^{N} w_{t+1}(n) = ∑_{n=1}^{N} e^{−y_{n} F_{t}(x_{n})} = ∑_{n=1}^{N} e^{−α_{t} y_{n} h_{t}(x_{n})} w_{t}(n) (t > 1)

(13.44)

Now we can find coefficient α_{t} in Eq. (13.40) as the one that minimizes the exponential loss E_{t+1}. To do so, we first separate each of the N terms in the summation above into two parts, corresponding to whether x_{n} is classified by h_{t} correctly or incorrectly:

E_{t+1} = ∑_{n=1}^{N} w_{t}(n)e^{−α_{t} y_{n} h_{t}(x_{n})} = ∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)e^{α_{t}} + ∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)e^{−α_{t}}

(13.45)

and then set its derivative with respect to α_{t} to zero:

\frac{dE_{t+1}}{dα_{t}} = ∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)e^{α_{t}} − ∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)e^{−α_{t}} = 0  (13.46)

Solving this equation we get α_{t} as the optimal coefficient that minimizes E_{t+1}:

α_{t} = \frac{1}{2} ln (\frac{∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)}{∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)}) = ln \sqrt{\frac{1 − ε_{t}}{ε_{t}}} > 0  (13.47)

which is greater than zero because ε_{t} < 1/2, 1 − ε_{t} > 1/2, and

e^{α_{t}} = \sqrt{\frac{1 − ε_{t}}{ε_{t}}} > 1, e^{−α_{t}} = \sqrt{\frac{ε_{t}}{1 − ε_{t}}} < 1  (13.48)

We see that if a weak classifier h_{t} has a small error ε_{t}, it will be weighted by a large α_{t} and therefore contributes more to the strong classifier H_{t}; on the other hand, if h_{t} has a large error ε_{t}, it will be weighted by a small α_{t} and contributes less to H_{t}. As the strong classifier H_{t} takes advantage of all previous weak classifiers h_{1}, ⋯, h_{t}, each of which may be more effective in a certain region of the N-D feature space than others, H_{t} can be expected to be a much more accurate classifier than any of the weak classifiers.

Replacing t by t + 1 in Eq. (13.43), we now get

w_{t+1}(n) = w_{t}(n) e^{−α_{t} y_{n}h_{t}(x_{n})} = \{ \begin{matrix}w_{t}(n)\sqrt{\frac{ε_{t}}{1−ε_{t}}} < w_{t}(n) & if y_{n}h_{t}(x_{n}) = 1 \\ w_{t}(n)\sqrt{\frac{1−ε_{t}}{ε_{t}}} > w_{t}(n) & if y_{n}h_{t}(x_{n}) = −1 \end{matrix}

(13.49)

We see that if y_{n}h_{t}(x_{n}) = 1, i.e., x_{n} is classified correctly by h_{t}, it will be weighted more lightly by w_{t+1}(n) < w_{t}(n) in the next iteration, but if y_{n}h_{t}(x_{n}) = −1, i.e., x_{n} is classified incorrectly by h_{t}, it will be weighted more heavily by w_{t+1}(k) > w_{t}(k) in the next iteration, thereby it will be emphasized and have a better chance to be corrected to have more accurately classified by h_{t+1} in the next iteration.

We further consider the ratio of the exponential losses of two consecutive iterations:

\frac{E_{t+1}}{E_{t}} = \frac{∑_{n=1}^{N} w_{t+1}(n)}{∑_{n=1}^{N} w_{t}(n)} = \frac{∑_{n=1}^{N} w_{t}(n)e^{−α_{t} y_{n} h_{t}(x_{n})}}{∑_{n=1}^{N} w_{t}(k)}
= \frac{∑_{y_{n}h_{t}(x_{n})=−1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}e^{α_{t}} + \frac{∑_{y_{n}h_{t}(x_{n})=1} w_{t}(n)}{∑_{n=1}^{N} w_{t}(n)}e^{−α_{t}}
= ε_{t} \sqrt{\frac{1 − ε_{t}}{ε_{t}}} + (1 − ε_{t}) \sqrt{\frac{ε_{t}}{1 − ε_{t}}} = 2\sqrt{ε_{t}(1 − ε_{t})} ≤ 1 (13.50)

This ratio is twice the geometric average \sqrt{ε_{t}(1 − ε_{t})} ≤ 1 of ε_{t} and 1 − ε_{t}, which reaches its maximum when ε_{t} = 1 − ε_{t} = 1/2. However, as ε_{t} < 1/2, the ratio is always smaller than 1. We see that the exponential cost can be approximated as an exponentially decaying function from its initial value E_{0} = ∑_{n=1}^{N} w_{0}(n) = N for some ε < 1/2:

lim_{t→∞} E_{t} ≈ lim_{t→∞} (2\sqrt{ε(1 − ε)})^{t} E_{0} = 0  (13.51)

We can therefore conclude that the exponential loss will always decrease, i.e., the error of AdaBoost will always converge to zero.

The weak classifier used in each iteration is typically implemented as a decision stump, a binary classifier that partitions the N-D feature space into two regions. Specifically, as a simple example, a coordinate descent method can be used by which all training samples are projected onto the ith dimension of the feature space d_{i}:

x_{n} = P_{d_{i}}(x_{n}) = \frac{x_{n}^{T} d_{i}}{||d_{i}||} , (n = 1, ⋯, N)  (13.52)

and then classified into two classes by a threshold T along the direction of d_{i}:

h(x_{n}) = \{ \begin{matrix}+1 & if x_{n} < T \\ −1 & if x_{n} > T \end{matrix}  (13.53)

The optimal decision stump is obtained when the following weighted error is minimized with respect to the threshold T along that direction of d_{i}:

ε_{t} = ∑_{n=1}^{N} w_{t}(n) δ(h_{t}(x_{n}) − y_{n}) = ∑_{h_{t}(x_{n})≠y_{n}} w_{t}(n)  (13.54)

Alternatively, the weak classifier can also be obtained based on the principal component analysis (PCA) by partitioning the N-D feature space along the directions of the eigenvectors of the between-class scatter matrix

S_{b} = \frac{1}{N} [N_{−}(m_{−1} − m)(m_{−1} − m)^{T} + N_{+}(m_{+1} − m)(m_{+1} − m)^{T} ] (13.55)

where N_{−} and N_{+} are the numbers of samples in the two classes (N_{−}+N_{+} = N), and

m_{±} = \frac{1}{N_{±}} ∑_{y_{n}=±1} w(n) x_{n}, m = \frac{1}{N} ∑_{n=1}^{N} w(n) x_{n}  (13.56)

are the weighted mean vectors of the two classes and the total weighted mean vector of all N samples in the training set. The training samples are likely to be better separated along these directions of the eigenvectors of S_{b}, which measures the separability of the two classes. The eigenequations of S_{b} is:

S_{b} = VΛV^{−1} = VΛV^{T}  (13.57)

As S_{b} is symmetric, the eigenvector matrix V is orthonormal and its columns can be used as an orthogonal basis that spans the feature space as well as the standard basis. The same binary threshold classification considered above for the weak classifiers can be readily applied along these PCA bases.

The Matlab code of the main iteration loop of the algorithm is listed below, followed by the essential functions called by the main loop. Here T is the maximum number of iteration, and N is the total number of training samples.

The algorithm can be carried out in the feature space based on either the PCA basis (columns of the eigenvector matrix V), or the original standard basis (columns of the identity matrix I).

The decision stump is implemented by a binary classifier, which partitions all training samples projected onto a 1-D space into two groups by a threshold value, which needs to be optimal in terms of the number of misclassification. A parameter plt is used to indicate the polarity of the binary classification, i.e., which of the two class C_{1} and C_{0} is on the lower (or higher) side of the threshold.

Plt=[];              % polarity of binary classification
Tr=[];               % transformation of basis vectors
Th=[];               % threshold of binary classification
Alpha=[];            % alpha values
h=zeros(T,N);        % h functions
F=zeros(T,N);        % F functions
w=ones(T,N);         % weights for N training samples
Er=N;                % error initialized to N
t=0;                 % iteration index
while t<T  & Er>0    % the main iteration
    t=t+1;
    [tr th plt er]=WeakClassifier(X,y,w(t,:),PCA);
                                % weak classifier
    alpha=log(sqrt((1-er)/er)); % update alpha
    Alpha=cat(1,Alpha,alpha);   % record alpha
    Tr=cat(2,Tr,tr');           % record transform vector
    Th=cat(1,Th,th);            % record threshold
    Plt=cat(1,Plt,plt);         % record polarity
    x=tr*X;                     % carry out transform
    c=sqrt(er/(1-er));
    for n=1:N                   % update weights
        h(t,n)=h_function(x(n),th,plt);  % find h function
       if h(t,n)*y(n)<0
           w(t+1,n)=w(t,n)/c;   % update weights
       else
           w(t+1,n)=w(t,n)*c;
       end
    end
    F(t,:)=Alpha(t)*h(t,:);     % get F functions
    if t>1
        F(t,:)=F(t,:)+F(t-1,:);
    end
    Er=sum(sign(F(t,:).*y)==-1);  % error of strong classifier,
                                  % number of misclassifications
    fprintf('%d: %d/%d=%f\n',t,Er,N,Er/N)
end

Here are the functions called by the main iteration loop above:

function [Tr,Th Plt Er]=WeakClassifier(X,y,w,pca)
% X: N columns each for one of the N training samples
% y: labeling of X
% w: weights for N training samples
% pca: use PCA dimension if pca~=0
% Er: minimum error among all D dimensions
% Tr: transform vector (standard or PCA basis)
% Th: threshold value
% Plt: polarity
[D N]=size(X);
n0=sum(y>0);            % number of samples in class C+
n1=sum(y<0);            % number of samples in class C-
if pca                  % find PCA basis
for i=1:D
    Y(i,:)=w.*X(i,:);
end
X0=Y(:,find(y>0));  % all samples in class C+
X1=Y(:,find(y<0));  % all samples in class C-
m0=mean(X0')';      % mean of C+
m1=mean(X1')';      % mean of C-
mu=(n0*m0+n1*m1)/N; % over all mean
Sb=(n0*(m0-mu)*(m0-mu)'+n1*(m1-mu)*(m1-mu)')/N;
                    % between-class scatter matrix
[v d]=eig(Sb);      % eigenvector and eigenvalue of Sb
else
v=eye(N);           % standard basis
end
Er=9e9;                 % initialize min error
for i=1:D               % for each of D dimensions
tr=v(:,i)';         % get an eigenvector of Sb
x=tr*X;             % rotate the vector
[th plt er]=BinaryClassifier(x,y,w);
                    % binary classify N samples in 1-D
er=0;
for n=1:N
    h(n)=h_function(x(n),th,plt);
                    % h-function of nth sample
    if h(n)*y(n)<0  % if misclassified
        er=er+w(n); % add error
    end
end
er=er/sum(w);       % total error of ith dimension

if er<Er            % a smaller error
    Er=er;          % update min error
    Plt=plt;        % polarity
    Th=th;          % threshold
    Tr=tr;          % transform vector
end
end
end
function h=h_function(x, th, plt)
if xor(x>th, plt)
h=1;
else
h=-1;
end
end
function [Th Plt Er]=BinaryClassifier(x,y,w)
N=length(x);
[x1 i]=sort(x);        % sort 1-D data x
y1=y(i);               % reorder targets
w1=w(i);               % reorder weights
Er=9e9;                & initialize min error
for n=1:N-1            % for N-1 ways of binary classification
e0=sum(w1(find(y1(1:n)==1)))+sum(w1(n+find(y1(n+1:N)==-1)));
e1=sum(w1(find(y1(1:n)~=1)))+sum(w1(n+find(y1(n+1:N)~=-1)));
if e1 > e0         % polarity: left -1, right +1
    plt=0; er=e0;
else               % polarity: left +1, right -1
    plt=1; er=e1;
end
if er<Er           % a smaller error
    Er=er;         % update min error
    Plt=plt;       % polarity
    Th=(x1(k)+x1(k+1))/2;  % threshold
end
end
end

Example 13.10 This example shows the classification of the two-class XOR dataset (used previously for testing the naive Bayes method), containing two classes of N = 200 samples each in 2-D space, represented by the red and blue dots in Fig. 13.13. The intermediate results after 4, 8, 16 and 32 iterations shows the progress of the iteration, when the error rate continuously reduces from 134/400, 85/400, 81/400, 58/100. Fig. 13.14 shows a sequence of 350 binary participations of the 2-D space (top left) and the final classification result (top right). The error rate eventually reduced to zero as guaranteed by the AdaBoost method. Although all training samples are correctly classified eventually, the result suffers the problem of overfitting.

Example 13.11 Fig. 13.15 shows the partitioning of the 2-D space by the AdaBoost algorithm trained on data samples from two concentric classes, along both the standard basis vectors (left) and the PCA directions (right). The PCA method performances significantly better in this example as its error reduces faster and it converges to zero after 61 iterations, while the error of the method based on the standard axes (in vertical and horizontal directions) does not go down to zero even after 120 iterations. This faster reduction of error of the PCA method can be easily understood as many different directions are used to partition the space into two regions, while in the coordinate descent method the partitioning of the space is limited to either of the two directions.

Fig. 13.16 shows how the AdaBoost algorithm is iteratively trained. The error rate of both the training and testing samples, are plotted as the iteration progresses (top). Also, the weighted error ε_{t}, the exponential costs E_{t}, and the ratio E_{t+1}/E_{t} = 2\sqrt{ε_{t}(1 − ε_{t})} are all plotted (bottom). We see that ε_{t} < 0.5 and E_{t+1}/E_{t} < 1 for all steps, and the exponential cost E_{t} attenuates exponentially towards zero.

Example 13.12 This example shows the classification of a dataset containing 200 samples of two classes (100 each) in an XOR pattern in the 2-D space, by the AdaBoost method based on both coordinate descent and PCA methods, as shown in Figs. 13.17 and 13.18, respectively. In both cases, half of the dataset (top-left) is used for training, while the other half for testing (top-right).

We see that the coordinate descent method performs poorly in both the slow convergence (more than 400 iterations) during training and high classification error rate (more than 1/3) during testing. The partitioning of the 2-D space is an obvious over fitting of the training set instead of reflecting the actual distribution of the two classes (four clusters). On the other hand, the PCA method converges quickly (10 iterations) and classifies the testing samples with very low error rate. The space is clearly partitioned into two regions corresponding to the two classes.

Problems

Carry out classification using both the iris and the handwritten digit datasets by each of the following three methods. In each case, cross validate your algorithm by using 50% randomly chosen samples in the dataset for training and the other 50% for testing. Show your classification results in the confusion matrix together with the error rate, the sum of all off-diagonal components of the confusion matrix divided by the total number of samples.

1. The K-nearest neighbor method

2. The minimum distance method based on Mahananobis distance

3. The naive Bayes method

328  Statistic Classification

Figure 13.13 Classification by AdaBoost on XOR Dataset (early iterations)

13.4 Adaptive Boosting 329

Figure 13.14 Classification by AdaBoost on XOR Dataset (final result)

330  Statistic Classification

Figure 13.15 AdaBoost Classification Along Basis Vectors and PCA Directions Figure 13.16 Progress of Iterative Training of AdaBoost

13.4 Adaptive Boosting 331

Figure 13.17 Classification of XOR Dataset (Coordinate Descent) Figure 13.18 Classification of XOR Dataset (PCA method)

14 Support Vector machine

14.1 Maximum Margin and Support Vectors

The support vector machine (SVM) is a supervised binary classifier trained by a data set {(x_{n}, y_{n}), n = 1, ⋯, N}, of which sample x_{n} belongs to either class C_{+} if labeled by y_{n} = 1 or class C_{−} if labeled by y_{n} = −1. The goal of the training process is to partition the feature space into two halves by a decision plane H_{0}, described by the following linear decision equation:

f(x) = w^{T} x + b = ∑_{i=1}^{d} x_{i}w_{i} + b = 0  (14.1)

in terms of its normal vector w and intercept b. Once these two parameters are determined in the training process, any unlabeled sample x can be classified into either of the two classes:

if f(x) = w^{T} x + b \{ \begin{matrix}> 0 \\ = 0 \\ < 0 \end{matrix} , then \{ \begin{matrix}x ∈ C_{+} \\ x ∈ H_{0} \\ x ∈ C_{−} \end{matrix}  (14.2)

Obviously the underlying assumption of this simple idea is the linear separability of the two classes. However, as we will see later, the SVM can also be applied to nonlinear and multiclass classifications.

We note that Eq. (14.1) is actually the same as Eq. (5.1) for linear regression, when the linear regression function f(x) is thresholded by the zero plane to become an equation f(x) = 0. This is illustrated in Fig. 14.1, which is essentially the same as Fig. 5.1. Although both SVM and linear regression methods need to obtain the optimal parameters w and b, they are very different in that the SVM, as a classification method, needs to determine w and b so that the resulting decision plane shall separate the training samples optimally, in the sense that its distance to the closest sample on either side, called support vectors, are maximized.

By moving the constant term in Eq. (14.1) to the right hand side and dividing both sides by ||x||, this equation can be rewritten as:

p_{w}(x) = \frac{w^{T} x}{||w||} = −\frac{b}{||w||}  (14.3)

where p_{w}(x) = w^{T} x/||w|| is the projection of any point x ∈ H_{0} on the decision

Figure 14.1 Binary Partition by Thresholding a Linear Function

plane H_{0} onto its normal direction w. The absolute value of this projection is the distance between H_{0} and the origin:

d(H_{0}, 0) = |p_{w}(x)| = \frac{|b|}{||w||}  (14.4)

We further find the projection of any point x′ ∉ H_{0} off the decision plane H_{0} onto w as p_{w}(x′) = w^{T} x′/||w||, and its distance to H_{0} is:

d(H_{0}, x′) = |p_{w}(x′) − p_{w}(x)| = |\frac{w^{T} x′}{||w||} − (−\frac{b}{||w||}) | = \frac{|w^{T} x′ + b|}{||w||} = \frac{|f(x′)|}{||w||}

(14.5)

Fig. 14.2 shows the projections of 2-D points, x ∈ H_{0}, together with x′ and x′′ on opposite sides of H_{0}, onto its normal direction w.

We desire to find the optimal decision plane H_{0} that separates the training samples belonging to the two different classes, assumed to be linearly separable, in such a way that its distance to the support vectors, denoted by x_{sv}, is maximized:

d(H_{0}, x_{sv} ∈ C_{+}) = d(H_{0}, x_{sv} ∈ C_{−}) = d(H_{0}, x_{sv}) = \frac{|f(x_{sv})|}{||w||}  (14.6)

We further consider two additional planes H_{+} and H_{−} that are in parallel with H_{0} and pass through the support vectors x_{sv} on either side of H_{0}:

f_{±}(x) = f(x) ± c = w^{T} x + b ± c = w^{T} x + b ± 1 = 0  (14.7)

Here for convenience we have set c = 1 as these two equations can be arbitrarily scaled. Based on Eq. (14.4), the distances from H_{+} or H_{−} to the origin can be

Figure 14.2 Decision Plane and Decision Margin

written as

d(H_{±}, 0) = \frac{|b ± 1|}{||w||}  (14.8)

and their distances to the decision plane H_{0}, called the decision margin, can be found as:

d(H_{±}, H_{0}) = |d(H_{±}, 0) − d(H_{0}, 0)| = |\frac{|b ± 1|}{||w||} − \frac{|b|}{||w||}| = \frac{1}{||w||}  (14.9)

To maximize this margin, we need to minimize ||w||.

For these planes to correctly separate all samples in the training set {(x_{n}, y_{n}) (n = 1, ⋯, N)}, they need to satisfy the following two conditions:

\{ \begin{matrix}f_{−}(x_{n}) = w^{T} x_{n} + b − 1 ≥ 0 & if y_{n} = 1 \\ f_{+}(x_{n}) = w^{T} x_{n} + b + 1 ≤ 0 & if y_{n} = −1 \end{matrix}  (n = 1, ⋯, N) (14.10)

where the equality is satisfied by the support vectors on H_{+} or H_{−}, while the inequalities are satisfied by all other samples farther away from H_{0} behind H_{+} or H_{−}. The two conditions above can be combined to become:

y_{n}(w^{T} x_{n} + b) ≥ 1, (n = 1, ⋯, N)  (14.11)

Now the task of finding the optimal decision plane H_{0} can be formulated as a constrained minimization problem:

minimize: \frac{1}{2}||w||^{2} = \frac{1}{2}w^{T} w
subject to: y_{n}(x_{n}^{T} w + b) ≥ 1, (n = 1, ⋯, N)  (14.12)

Figure 14.3 Decision Plane and Margin

Figure 14.4 Decision Plane and Support Vectors

Fig. 14.3 illustrates the three planes considered above in 2-D case, including the decision plane H_{0}, together with H_{+} and H_{−}, as well as the decision margin 1/||w||.

Example 14.1 In Fig. 14.4, the straight line in 2-D space, denoted by H_{0}, is described by the following linear equation

f(x) = w^{T} x + b = [w_{1}, w_{2}] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] + b = [1, 2] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] − 1 = x_{1} + 2x_{2} − 1 = 0

(14.13)

The distance from H_{0} to the origin is:

d(H_{0}, 0) = \frac{|b|}{||w||} = \frac{1}{\sqrt{w_{1}^{2} + w_{2}^{2}}} = \frac{1}{\sqrt{1^{2} + 2^{2}}} = \frac{1}{\sqrt{5}} = 0.447  (14.14)

which is invariant with respect to any scaling of the equation. Consider three points in the space:

• x_{0} = [0.5, 0.25]^{T}, f(x_{0}) = 0.5 + 2 × 0.25 − 1 = 0, i.e., x_{0} is on the plane. Its distance to the plane is d(x_{0}, H) = f(x_{0})/||w|| = 0.

• x_{1} = [1, 0.25]^{T}, f(x_{1}) = 1 + 2 × 0.25 − 1 = 0.5 > 0, i.e., x_{1} is above the straight line, its distance to the plane is d(x_{1}, H) = f(x_{1})/||w|| = |0.5|/\sqrt{5} = 0.2235.

• x_{2} = [0.5, 0]^{T}, f(x_{2}) = 0.5+2×0−1 = −0.5 < 0, i.e., x_{2} is below the straight line, its distance to the plane is d(x_{2}, H) = f(x_{2})/||w|| = | − 0.5|/\sqrt{5} = 0.2235.

As the two points x_{1} = [1, 0.25]^{T} and x_{2} = [0.5, 0]^{T} have equal distance to H_{0}, the straight line f(x) = x_{1} + 2x_{2} − 1 = 0, we assume they are two support vectors x_{sv} on either side of H_{0}:

|f(x_{sv})| = |w^{T} x_{1} + b| = |w^{T} x_{2} + b| = 0.5  (14.15)

Based on the decision equation scaled by 2:

f(x) = w^{T} x + b = 2x_{1} + 4x_{2} − 2 = 0  (14.16)

we further get the equations for the two planes H_{+} and H_{−} both parallel o H_{0}:

H_{+} : 2x_{1} + 4x_{2} − 2 − 1 = 2x_{1} + 4x_{2} − 3 = 0
H_{−} : 2x_{1} + 4x_{2} − 2 + 1 = 2x_{1} + 4x_{2} − 1 = 0

Their distances to the origin are

d(H_{+}, 0) = \frac{| − 3|}{||w||} = 1.341, d(H_{−}, 0) = \frac{| − 1|}{||w||} = 0.447  (14.17)

and the distance between them is

d(H_{−}, H_{+}) = d(H_{+}, 0) − d(H_{−}, 0)1.341 − 0.447 = 0.894 = \frac{2}{||w||} (14.18)

which is indeed twice the distance d(H_{±}, H_{0}) = 1/||w|| = 0.447:

For reasons to be discussed later, instead of directly solving the constrained minimization problem in Eq. (14.12), now called the primal problem, we actually solve the dual problem (Section 3.3). We first construct the Lagrangian function of the primal problem, called the primal function:

L_{p}(w, b, α) = \frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))  (14.19)

where α_{1}, ⋯, α_{N} are the Lagrange multipliers. For this minimization problem with non-positive constraints, the Lagrangian multipliers are required to be negative, α_{n}, according to Table 3.21 in Section 3.2, if a minus sign is used for the second term. However, to be consistent with all SVM literatures, we use the positive sign for the second term and require α_{n} ≥ 0. Note that if x_{n} is a support vector on either H_{+} or H_{−}, i.e., the equality constraint y_{n}(w^{T} x_{n} + b) = 1 holds, then α_{n} > 0; but if x_{n} is not a support vector, the equality constraint does not hold, and α_{n} = 0.

As discussed in Subsection 3.3, we can solve this constrained optimization problem by solving an equivalent dual problem, i.e., we first find the lower bound of L_{p}(w, b, α) with respect to w and b, and then find the tightest bound with respect to α). Such a solution has to satisfy the KKT conditions (Eqs. (3.45) through (3.48):

• Stationarity:

\frac{∂}{∂b}L_{p}(w, b) = \frac{∂}{∂b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))]

= ∑_{n=1}^{N} α_{n}y_{n} = 0  (14.20)

and

\frac{∂}{∂w} L_{p}(w, b) = \frac{∂}{∂w} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))]

= w − ∑_{n=1}^{N} α_{n}y_{n}x_{n} = 0,  (14.21)

i.e.,

w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}  (14.22)

• Primal feasibility:

y_{n}(w^{T} x + b) − 1 ≥ 0, (n = 1, ⋯, N)  (14.23)

• Dual feasibility:

α_{n} ≥ 0, (n = 1, ⋯, N)  (14.24)

• Complementarity:

α_{n}[y_{n}(w^{T} x + b) − 1] = 0  (14.25)

To find the minimum as the lower bound of the primal function L_{p}(w, b) in Eq. (14.19), we substitute Eqs. (14.20) and (14.22) back into the primal function, we get the dual function:

L_{d}(α) = min_{w,b} L_{p}(w, b, α) = min_{w,b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n}(1 − y_{n}(w^{T} x_{n} + b))]

= min_{w,b} [\frac{1}{2}w^{T} w + ∑_{n=1}^{N} α_{n} − w^{T} ∑_{n=1}^{N} α_{n}y_{n}x_{n} − b ∑_{n=1}^{N} α_{n}y_{n}]

= \frac{1}{2} ( ∑_{m=1}^{N} α_{m}y_{m}x_{m})^{T} (∑_{n=1}^{N} α_{n}y_{n}x_{n})

+ ∑_{n=1}^{N} α_{n} − ( ∑_{m=1}^{N} α_{m}y_{m}x_{m})^{T} (∑_{n=1}^{N} α_{n}y_{n}x_{n})

= ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m})  (14.26)

For this dual function L_{d}(α) to be the tightest lower bound of the primal function, it needs to be maximized with respect to α, subject to the constraint imposed by Eq. (14.20):

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m} (x_{n}^{T} x_{m}) = 1^{T} α − \frac{1}{2}α^{T} Qα

subject to: ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N)  (14.27)

where Q is an N by N symmetric matrix of which the component in the mth row and nth column is Q(m, n) = y_{m}y_{n}x_{m}^{T}x_{n} (m, n = 1, ⋯, N). Now we have converted the primal problem of constrained minimization of the primal function L_{p}(w, b, α) with respect to w and b to its dual problem of linearly constrained maximization of the dual function L_{d}(α) with respect to {α_{1}, ⋯, α_{N}}. Solving this quadratic programming (QP) problem (Section 3.6) we get all Lagrange multipliers α_{n} ≥ 0, (n = 1, ⋯, N). All training samples x_{n} corresponding to positive Lagrange multipliers α_{n} > 0 are support vectors, while others corresponding to α_{n} = 0 are not support vectors.

We can now find the solution w^{∗} based on Eq. (14.22):

w^{∗} = ∑_{n=1}^{N} α_{n}y_{n}x_{n} = ∑_{n∈sv} α_{n}y_{n}x_{n} = ∑_{x_{n}∈C_{+}, n∈sv} α_{n}x_{n} − ∑_{x_{n}∈C_{−}, n∈sv} α_{n}x_{n}

(14.28)

Note that the normal vector is the difference between the weighted means of the support vectors in class C_{+} (first term) and class C_{−} (second term), and it is determined only by the support vectors.

We can further find b^{∗} based on Eq. (14.25) for complementarity of the KKT conditions. As the Lagrangian multiplier α_{n} corresponding to any support vector x_{n} is non-zero, we must have y_{n}(w^{T} x_{n}+b)−1 = 0, which is actually the equality of Eq. (14.11) satisfied by all support vectors:

y_{n} (w^{T} x_{n} + b) = 1 or w^{T} x_{n} + b = y_{n}, (n ∈ sv)  (14.29)

(as y_{n}^{2} = 1). Solving for b we get the optimal b^{∗}:

b^{∗} = y_{n} − w^{T} x_{n} = y_{n} − ∑_{m∈sv} α_{m}y_{m}(x_{m}^{T}x_{n}), (n ∈ sv)  (14.30)

All support vectors should yield the same result. Computationally we simply get b^{∗} as the average of the above for all support vectors.

We note that both w and b depend only on the support vectors on planes H_{+} and H_{−}, corresponding to a positive α_{n} > 0, while all remaining samples corresponding to α_{n} = 0 are farther away from the decision plane H_{0}, behind either H_{+} or H_{−}. We see that the SVM is solely based on those support vectors in the training set, once they are identified during the training process, all other non-support vectors are irrelevant.

Once w and b are obtained in the training process, any unlabeled sample x can be classified into either of the two classes depending on whether the following decision function is greater or smaller than zero:

f(x) = w^{T} x + b = ( ∑_{n∈sv} α_{n}y_{n}x_{n})^{T} x + b

= ∑_{n∈sv} α_{n}y_{n} (x_{n}^{T} x) + b \{ \begin{matrix}> 0 & x ∈ C_{+} \\ < 0 & x ∈ C_{−} \end{matrix}  (14.31)

We also get:

||w||^{2} = w^{T} w = ( ∑_{n∈sv} α_{n}y_{n}x_{n}^{T} ) ( ∑_{m∈sv} α_{m}y_{m}x_{m})
= ∑_{n∈sv} α_{n}y_{n} ∑_{m∈sv} α_{m}y_{m}(x_{n}^{T} x_{m})
= ∑_{n∈sv} α_{n}y_{n}(y_{n} − b) = ∑_{n∈sv} α_{n}(1 − y_{n}b)
= ∑_{n∈sv} α_{n} − b ∑_{n∈sv} α_{n}y_{n} = ∑_{n∈sv} α_{n}  (14.32)

The last equality is due to equality constraints of the dual problem ∑_{n=1}^{N} α_{n}y_{n} = 0. Now the decision margin can be written as:

\frac{1}{||w||} = ( ∑_{n∈sv} α_{n})^{−1/2}  (14.33)

The SVM algorithm for binary classification can now by summarized as the following steps:

• Find the Lagrange multipliers {α_{1}, ⋯, α_{N}} by solving the QP problem in Eq. (14.27);

• Find the normal vector w by Eq. (14.28);

• Find the bias b by Eq. (14.30);

• Classify unlabeled x by Eq. (14.31).

As the last expression of the decision function in Eq. (14.31) depends only on α_{n} as well as x_{n} and y_{n} in the training set, the normal vector w is never needed, and the second step above for calculating w by Eq. (14.28) can be dropped.

We prefer to solve this dual problem not only because it is easier than the original primal problem, but also, more importantly, for the reason that all data points appear only in the form of an inner product x_{n}^{T} x_{m} in the dual problem, allowing the kernel method to be used, as discussed later.

The Matlab code for the essential part of the algorithm is listed below, where X and y are respectively the data array composed of N training vectors [x_{1}, ⋯, x_{N}] and their corresponding labelings y_{1}, ⋯, y_{N}, and IPmethod is a function that implements the interior point method (Section 3.7) for solving a general QP problem of minimizing x^{T} Qx/2 + c^{T} x subject to the linear equality constraints Ax = b and x ≥ 0.

    [X y]=getData;       % get the training data
    [m n]=size(X);
    Q=(y'*y).*(X'*X);    % compute the quadratic matrix
    c=-ones(n,1);        % coefficients of linear term
    A=y;                 % coefficient matrix
    b=0;                 % constants of linear equality constraints
    alpha=0.1*ones(n,1); % initial guess of solution
    [alpha mu lambda]=IPmethod(Q,A,c,b,alpha);
                         % solve QP for alpha (interior point method)
    I=find(abs(alpha)>10^(-3));  % indices of non-zero alphas
    asv=alpha(I);        % non-zero alphas
    Xsv=X(:,I);          % support vectors
    ysv=y(:,I);          % their labels
    w=sum(repmat(ysv.*asv,m,1).*Xsv,2);  % normal vector (not needed)
    bias=mean(ysv-w'*Xsv);               % bias

Example 14.2 The training set contains two classes of 2-D points with Gaussian distributions generated based on the following mean vectors and covariance matrices:

m_{−} = [ \begin{matrix}3.5 \\ 0 \end{matrix} ] , S_{−} = [ \begin{matrix}1 & 0.5 \\ 0.5 & 1 \end{matrix} ] , m_{+} = [ \begin{matrix}0 \\ 3.5 \end{matrix} ] , S_{+} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ]

The SVM algorithm finds the weight vector w and constant b of the optimal boundary between the two classes based on three support vectors, all listed

Figure 14.5 A Simple Example of Binary Classification by SVM

below. Note that decision boundary is completely dictated by the three support vectors and the margin distance between them is maximized.

\begin{matrix}n & α_{n} & x_{n} = [x_{1}, x_{2}] & y_{n} \\ 40 & 0.52 & x = [4.43, 2.44] & −1 \\ 52 & 3.11 & x = [1.17, 0.74] & −1 \\ 103 & 3.64 & x = [1.30, 1.64] & 1 \end{matrix} ,  w = [ \begin{matrix}−1.25 \\ 2.39 \end{matrix} ] ,  b = −1.31

The binary classification result is shown in Fig. 14.5.

14.2 Kernel Mapping

The SVM algorithm above converges only if the data points of the two classes in the training set are linearly separable. If this is not the case, we can consider the kernel method first introduced in Section 11.1.

Kernel mapping can be applied to the SVM algorithm as all data points appear in the algorithm are in the form of an inner product. Specifically, during the training process, we replace the inner product x_{m}^{T}x_{n} in both Eq. (14.27) and Eq. (14.30) by the kernel function K(x_{m}, x_{n}) to get

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m}K(x_{n}, x_{m})

subject to: ∑_{n=1}^{N} α_{n}y_{n} = y^{T} α = 0, α_{n} ≥ 0 (n = 1, ⋯, N) (14.34)

and

b = y_{n} − ∑_{m∈sv} α_{m}y_{m}K(x_{m}, x_{n}), (n ∈ sv)  (14.35)

We also replace the inner product x_{n}^{T} x in Eq. (14.31) by K(x_{n}, x) for the classification of any unlabeled point z = ϕ(x):

f(z) = w^{T} z+b = ∑_{n∈sv} α_{n}y_{n} (z_{n}^{T} z)+b = ∑_{n∈sv} α_{n}y_{n} K(x_{n}, x)+b \{ \begin{matrix}> 0 & x ∈ C_{+} \\ < 0 & x ∈ C_{−} \end{matrix}

(14.36)

Again, we note that the normal vector w in Eq. (14.28) never needs to be explicitly calculated. As now both the training and classification are carried out in some higher dimensional space z = ϕ(x), in which the classes are more likely linearly separable, the classification can be more effectively. More generally, the kernel method can be applied to any algorithm, so long as the data always appear in the form of an inner product.

14.3 Soft Margin SVM

When the two classes C_{−} and C_{+} are not linearly separable, the condition for the optimal hyperplane in Eq. (14.11) can be relaxed by including a slack variable ξ_{n} ≥ 0 representing an extra error term:

y_{n}(x_{n}^{T} w + b) ≥ 1 − ξ_{n}, (n = 1, ⋯, N)  (14.37)

For better separate data points belonging to different classes, this error ξ_{n} needs to be minimized as well as ||w||. Now the primal problem in Eq. (14.12) can be modified to the following minimization problem with an objective function and two sets of inequality (non-negative) constraints:

minimize: \frac{1}{2}||w||^{2} + C ∑_{n=1}^{N} ξ_{n}
subject to: y_{n}(x_{n}^{T} w + b) + ξ_{n} − 1 ≥ 0, ξ_{n} ≥ 0; (n = 1, ⋯, N)(14.38)

The value of the parameter C can be adjusted to make a proper tradeoff between under fitting emphasizing the global distribution of the training samples, and over fitting emphasizing the local samples close to the decision boundary H_{0}. A large C value emphasizes the minimization of the error term in the objective function, and the resulting H_{0} is mostly dictated by a small number of local support vectors in the region between the class, including possibly some noisy outliers, thereby risking the problem of overfitting. On the other hand, a small C value de-emphasizes the error term and the classifier can tolerate greater errors, thereby allowing more data samples farther away from H_{0} to become support vectors and play a role in determining H_{0}. Consequently the classification trends to better reflect the overall distribution of the data samples, but with the risk of overfitting.

This constrained minimization problem can be converted to the minimization of the corresponding Lagrangian:

L_{p}(w, b, ξ, α, μ) = \frac{1}{2}||w||^{2} + C ∑_{n=1}^{N} ξ_{n} − ∑_{n=1}^{N} α_{n}[y_{n}(w^{T} x + b) + ξ_{n} − 1] − ∑_{n=1}^{N} β_{n}ξ_{n}

(14.39)

where β_{n} as well as α_{n} is the Lagrange multipliers for the two sets of constraints. As they are non-negative (instead of non-positive as in previous section), minus sign is used in front of the last two terms for the constraints, so that the Lagrange multipliers α_{n} and β_{n} are still required to be positive (in consistent with Table 3.21). The KKT conditions (Eqs. (3.45) through (3.48)) of this minimization problem are listed below (for all n = 1, ⋯, N):

• Stationarity:

\frac{∂L_{p}}{∂w} = w − ∑_{n=1}^{N} y_{n}α_{n}x_{n} = 0, i.e. w = ∑_{n=1}^{N} y_{n}α_{n}x_{n}; (14.40)

\frac{∂L_{p}}{∂b} = ∑_{n=1}^{N} y_{n}α_{n} = 0;  (14.41)
\frac{∂L_{p}}{∂ξ_{n}} = C − α_{n} − β_{n} = 0;  (14.42)

• Primal feasibility:

y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1 ≥ 0, ξ_{n} ≥ 0,  (14.43)

• Complementarity:

α_{n}[y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1] = 0, β_{n}ξ_{n} = 0,  (14.44)

• Dual feasibility:

α_{n} ≥ 0, β_{n} ≥ 0  (14.45)

Depending on the value of α_{n}, there exist three possible cases, due to the complementarity (Eq. (14.44)):

• If α_{n} = 0, then β_{n} = C − α_{n} = C > 0, ξ_{n} = 0, and we have

y_{n}(w^{T} x_{n} + b) − 1 ≥ 0  (14.46)

• If 0 < α_{n} < C, i.e., α_{n} ≠ 0, then we have

y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1 = 0  (14.47)

But as β_{n} = C − α_{n} > 0, ξ_{n} = 0 and the equation above becomes

y_{n}(w^{T} x_{n} + b) − 1 = 0  (14.48)

• If α_{n} = C ≠ 0, then we have

y_{n}(w^{T} x_{n} + b) + ξ_{n} − 1 = 0  (14.49)

But as β_{n} = C − α_{n} = 0, ξ_{n} ≥ 0 and the equation above becomes

y_{n}(w^{T} x_{n} + b) − 1 ≤ 0  (14.50)

In all three cases the expression y_{n}(w^{T} x_{n} + b) − 1 can be rewritten as

y_{n}(w^{T} x_{n} + b) − 1 = y_{n}(w^{T} x_{n} + b) − y_{n}^{2} = y_{n}[(w^{T} x_{n} + b) − y_{n}] = y_{n} E_{n} (14.51)

where E_{n} = (w^{T} x_{n} + b) − y_{n} is the error, or the difference between the actual and desired output.

We summarize all three cases above to get:

y_{n} E_{n} \{ \begin{matrix}≥ 0 & if α_{n} = 0 \\ = 0 & if 0 < α_{n} < C \\ ≤ 0 & if α_{n} = C \end{matrix}  (14.52)

These results derived from the KKT conditions in Eqs. (14.40) through (14.45) are an alternative form of the KKT conditions, which will be used in SMO algorithm to check if the KKT conditions are violated by any α_{n}.

The dual of the primal problem in Eq. (14.39) can be obtained by substituting the first set of equations (stationarity) into the primal Lagrangian:

maximize: L_{d}(α) = ∑_{n=1}^{N} α_{n} − \frac{1}{2} ∑_{n=1}^{N} ∑_{m=1}^{N} α_{n}α_{m}y_{n}y_{m}x_{m}^{T}x_{n}

+ C ∑_{n=1}^{N} ξ_{n} − ∑_{n=1}^{N} α_{n}ξ_{n} − ∑_{n=1}^{N} β_{n}ξ_{n}

= 1^{T} α − \frac{1}{2}α^{T} Qα + C ∑_{n=1}^{N} ξ_{n} − ∑_{n=1}^{N} α_{n}ξ_{n} − ∑_{n=1}^{N} β_{n}ξ_{n}

subject to: 0 ≤ α_{n} ≤ C, ∑_{n=1}^{N} α_{n}y_{n} = 0  (14.53)

Note that due to the condition C = α_{n}+β_{n}, the Lagrangian multipliers ξ_{n} and β_{n} no longer appear in the Lagrangian L_{d} of the dual problem, which happens to be identical to that of the linearly separable problem discussed in previous section. Also, since α_{n} ≥ 0, β_{n} ≥ 0 and C = α_{n} + β_{n}, we have 0 ≤ α_{n} = C − β_{n} ≤ C. Having solved this QP problem for α_{1}, ⋯, α_{N}, we get the optimal decision plane in terms of the normal vector of the decision plane based on Eq. (14.28):

w = ∑_{n=1}^{N} y_{n}α_{n}x_{n} = ∑_{n∈sv} y_{n}α_{n}x_{n}  (14.54)

and the bias term b based on Eq. (14.30):

b = y_{n} − ∑_{i∈sv} y_{i}α_{i}x_{i}^{T} x_{n} = y_{n} − ∑_{i∈sv} y_{i}α_{i}K(x_{i}, x_{n}), (n ∈ sv)  (14.55)

where the inner product x_{k}^{T} x is replaced by the kernel function K(x_{k}, x) if it is nonlinear. Computationally, we take the average of such b values of all support vectors as the offset term.

Once w and b are obtained, any unlabeled point x can be classified into either of the two classes depending on whether it is on the positive or negative side of the decision plane:

w^{T} x + b = ∑_{i∈sv} y_{i}α_{i}x_{i}^{T} x + b = ∑_{i∈sv} y_{i}α_{i}K(x_{i}, x) + b \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix} (14.56)

where the inner product is replaced by a kernel function if it is nonlinear.

14.4 Sequential Minimal Optimization Algorithm

The sequential minimal optimization (SMO) is a more efficient algorithm for solving the SVM problem, compared with the generic QP algorithms such as the interior-point method. The SMO algorithm can be considered as a method of decomposition, by which an optimization problem of multiple variables is decomposed into a series of sub-problems each optimizing an objective function of a small number of variables, typically only one, while all other variables are treated as constants that remain unchanged in the sub-problem. For example, the coordinate descent algorithm is just such a decomposition method, which solves a problem in a multidimensional space by converting it into a sequence of sub-problems each in a one-dimensional space.

When applying the decomposition method to the soft margin SVM problem, we could consider optimizing only one variable α_{i} at a time, while treating all remaining variables α_{1}, ⋯, α_{i−1}, α_{i+1}, ⋯, α_{N} as constants. However, due to the constraint ∑_{n=1}^{N} α_{n}y_{n} = 0, α_{i} is a linear combination of N − 1 constants and is therefore also a constant. We therefore need to consider two variables at a time, denoted by α_{i} and α_{j}, while treating the remaining N − 2 variables as constants. The objective function of such a sub-problem can be obtained by dropping all constant terms independent of the two selected variables α_{i} and α_{j} in the original objective function in Eq. (14.53):

maximize: L(α_{i}, α_{j}) = α_{i} + α_{j} − \frac{1}{2} (α_{i}^{2}x_{i}^{T} x_{i} + α_{j}^{2}x_{j}^{T} x_{j} + 2α_{i}α_{j}y_{i}y_{j}x_{i}^{T} x_{j})

−α_{i}y_{i} (∑_{n≠i} α_{n}y_{n}x_{n}^{T} ) x_{i} − α_{j}y_{j} (∑_{n≠j} α_{n}y_{n}x_{n}^{T} ) x_{j}

= α_{i} + α_{j} − \frac{1}{2} (α_{1}^{2}K_{ii} + α_{2}^{2}K_{jj} + 2α_{i}α_{j}y_{i}y_{j}K_{ij})

−α_{i}y_{i} ∑_{n≠i,j} α_{n}y_{n}K_{ni} − α_{j}y_{j} ∑_{n≠i,,j} α_{n}y_{n}K_{nj}

subject to: 0 ≤ α_{i}, α_{j} ≤ C, ∑_{n=1}^{N} α_{n}y_{n} = 0  (14.57)

Here all inner products x_{m}^{T}x_{n} are replaced by a kernel the corresponding kernel function K_{mn} = K(x_{m}, x_{n}).

This maximization problem can be solved iteratively. Out of all previous values α_{n}^{old} (n = 1, ⋯, N), the two selected variables are updated to become α_{i}^{new} and α_{j}^{new} by which L(α_{i}, α_{j}) is maximized subject to the two constraints.

We rewrite the second constraint as

α_{i}y_{i} + α_{j}y_{j} = − ∑_{n≠i,j} α_{n}y_{n}  (14.58)

and multiply both sides by y_{i} to get

y_{i}^{2}α_{i} + y_{i}y_{j}α_{j} = α_{i} + sα_{j} = (− ∑_{n≠i,j} α_{n}y_{n}) y_{i} = δ, i.e. α_{i} = δ − sα_{j}

(14.59)

where y_{i}^{2} = 1, and we have defined

s = y_{i}y_{j},  δ = − ∑_{n≠i,j} α_{n}y_{n}y_{i}  (14.60)

As δ is independent of α_{i} and α_{j}, it remains the same before and after they are updated, and we have

α_{i}^{new} + sα_{j}^{new} = α_{i}^{old} + sα_{j}^{old} = δ  (14.61)

i.e.,

Δα_{i} = α_{i}^{new} − α_{i}^{old} = −s(α_{j}^{new} − α_{j}^{old}) = −sΔα_{j}  (14.62)

We now consider a closed-form solution for updating α_{i} and α_{j} in each iteration of this two-variable optimization problem. We first rewrite Eq. (14.28) as

∑_{n≠i,j} α_{n}y_{n}x_{n} = w − α_{i}y_{i}x_{i} − α_{j}y_{j}x_{j}  (14.63)

so that the two summations in L(α_{i}, α_{j}), now denoted by v_{k} (k = i, j), can be written as

v_{k} = ∑_{n≠i,j} α_{n}y_{n}K_{nk} = ∑_{n=1}^{N} α_{n}y_{n}K_{nk} − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}

= (∑_{n=1}^{N} α_{n}y_{n}K_{nk} + b) − b − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}
= u_{k} − b − α_{i}y_{i}K_{ik} − α_{j}y_{j}K_{jk}  (14.64)

where

u_{k} = f(z_{k}) = ∑_{n=1}^{N} α_{n}y_{n}K_{nk} + b  (14.65)

is the output in Eq. (14.36) corresponding to input x_{k}. Now the objective function above can be written as:

L(α_{i}, α_{j}) = α_{i} + α_{j} − \frac{1}{2}(α_{i}^{2}K_{ii} + α_{j}^{2}K_{jj} + 2sα_{i}α_{j}K_{ij}) − α_{i}y_{i}v_{i} − α_{j}y_{j}v_{j} (14.66)

Substituting α_{i} = δ − sα_{j} (Eq. (14.62)) into L(α_{i}, α_{j}), we can rewrite it as a function of a single variable α_{j} alone:

L(α_{j}) = δ + (1 − s)α_{j} − \frac{1}{2} [(δ − sα_{j})^{2}K_{ii} + α_{j}^{2}K_{jj} + 2s(δ − sα_{j})α_{j}K_{ij}]
−(δ − sα_{j})y_{i}v_{i} − α_{j}y_{j}v_{j}
= \frac{1}{2}(2K_{ij} − K_{ii} − K_{jj})α_{j}^{2}
+[1 − s + sδ(K_{ii} − K_{ij}) + y_{j}(v_{i} − v_{j})]α_{j} + Const  (14.67)

where s^{2} = (y_{i}y_{j})^{2} = 1, and scalar Const represents all constant terms independent of α_{i} and α_{j} and can therefore be dropped. Now the objective function becomes a quadratic function of the single variable α_{j}. Consider the coefficients of this quadratic function:

• The coefficient of α_{j}^{2}:

\frac{1}{2}(2K_{ij} − K_{ii} − K_{jj}) = \frac{1}{2}η  (14.68)

where η is defined as

η = 2K_{ij} − K_{ii} − K_{jj} = 2z_{i}^{T} z_{j} − z_{i}^{T} z_{i} − z_{j}^{T} z_{j} = −(z_{i} − z_{j})^{T} (z_{i} − z_{j})
= −||z_{i} − z_{j}||^{2} ≤ 0  (14.69)

• The coefficient of α_{j}:

1 − s + sδ(K_{ii} − K_{ij}) + y_{j}(v_{i} − v_{j})
= 1 − s + (sα_{i} + α_{j})(K_{ii} − K_{ij})

+y_{j}[(u_{i} − b − α_{i}y_{i}K_{ii} − α_{j}y_{j}K_{ij}) − (u_{j} − b − α_{i}y_{i}K_{ij} − α_{j}y_{j}K_{jj})]

= 1 − s + sα_{1}(K_{ii} − K_{ij}) + α_{2}(K_{ii} − K_{ij})
+y_{j}(u_{i} − u_{j}) − α_{i}sK_{ii} − α_{j}K_{ij} + α_{i}sK_{ij} + α_{j}K_{jj}
= y_{j}^{2} − s − α_{j}(2K_{ij} − K_{ii} − 2K_{jj}) + y_{j}(u_{i} − u_{j})
= y_{j}(u_{i} − y_{j} − u_{j} + y_{j}) − α_{j}(2K_{ij} − K_{ii} − K_{jj})
= y_{j}(E_{i} − E_{j}) − α_{j}η  (14.70)

where we have defined

E_{k} = u_{k} − y_{k} = ∑_{n=1}^{N} α_{n}^{old}y_{n}K_{nk} + b − y_{k} (k = 1, 2)  (14.71)

as the difference between the desired output y_{k} and the actual output based on the previous values of the variables α_{n}^{old} (n = 1, ⋯, N).

Now the quadratic objective function can be rewritten as:

L(α_{j}) = \frac{1}{2}ηα_{j}^{2} + [y_{j}(E_{i} − E_{j}) − α_{j}^{old}η] α_{j}  (14.72)

Given the previous values α_{n}^{old} (n = 1, ⋯, N) in the coefficients, we will find α_{j}^{new} that maximizes L(α_{j}). Consider its first and second order derivatives:

\frac{d}{dα_{j}} L(α_{j}) = ηα_{j} + y_{j}(E_{i} − E_{j}) − α_{j}^{old}η
\frac{d^{2}}{dα_{j}^{2}} L(α_{j}) = η ≤ 0  (14.73)

As the second order derivative of L(α_{j}) is η ≤ 0, it has a maximum. Solving the equation d L(α_{j})/dα_{j} = 0, we get α_{j}^{new} that maximizes L(α_{j}) based on the old value α_{j}^{old}:

α_{j}^{new} = α_{j}^{old}+\frac{y_{j}(E_{j} − E_{i})}{η} = α_{j}^{old}+Δα_{j}, Δα_{j} = α_{j}^{new}−α_{j}^{old} = \frac{y_{j}(E_{j} − E_{i})}{η},

(14.74)

Note that this is an unconstrained solution, which needs to be modified if the constraints in Eq. (14.57) are violated. As shown in Fig. 14.6, α_{i} and α_{j} are inside the square of size C, due to the first constraint 0 ≤ α_{i}, α_{j} ≤ C; also, due to the second constraint α_{i} + sα_{j} = δ (Eq. (14.59)), they have to be on the diagonal line segment inside the square.

The panels in Fig. 14.6 illustrate the four possible cases depending on whether s = 1 (y_{i} = y_{j} = ±1, panels 1 and 2), or s = −1 (y_{i} = −y_{j} = ±1, panels 3 and 4), and whether δ < C (panels 1 and 3) or δ > C (panels 2 and 4), which can

Figure 14.6 Four Cases while Updating α_{i} and α_{i}

be further summarized below in terms of the lower bound L and upper bound H of α_{2}:

• If s = 1, then α_{i} + sα_{j} = α_{i} + α_{j} = δ, – if δ < C, then min(α_{j}) = 0, max(α_{j}) = δ (first panel)

– if δ > C, then min(α_{j}) = δ − C, max(α_{j}) = C (second panel)

Combining these two cases, we have

\{ \begin{matrix}L = max(0, δ − C) = max(0, α_{i} + α_{j} − C) \\ H = min(δ, C) = min(α_{i} + α_{j}, C) \end{matrix}  (14.75)

• If s = −1, then α_{i} + sα_{j} = α_{i} − α_{j} = δ, – if δ > 0, then min(α_{j}) = 0, max(α_{j}) = C − δ (third panel)

– if δ < 0, then min(α_{j}) = −δ, max(α_{j}) = C (fourth panel)

Combining these two cases, we have

\{ \begin{matrix}L = max(0, −δ) = max(0, α_{j} − α_{i}) \\ H = min(C − δ, C) = min(C + α_{j} − α_{i}, C) \end{matrix}  (14.76)

Now we apply the constraints in terms of L and H to t optimal solution α_{j}^{new} obtained previously to get α_{j} that is feasible as well as optimal:

α_{j}^{new} ⟸ \{ \begin{matrix}H & if α_{j}^{new} ≥ H \\ α_{j}^{new} & if L < α_{2}^{new} < H \\ L & if α_{j}^{new} ≤ L \end{matrix}  (14.77)

We can further find α_{i} based on Eq. (14.62):

α_{i}^{new} = α_{i}^{old} − sΔα_{j} = α_{i}^{old} − s(α_{j}^{new} − α_{j}^{old})  (14.78)

and update the weight vector based on Eq. (14.40):

w^{new} = ∑_{n≠i,j} y_{n}α_{n}^{old}x_{n} + y_{i}α_{i}^{new}x_{i} + y_{j}α_{j}^{new}x_{j}
= w^{old} − y_{i}α_{i}^{old}x_{i} − y_{j}α_{j}^{old}x_{j} + y_{i}α_{i}^{new}x_{i} + y_{j}α_{j}^{new}x_{j}
= w^{old} + y_{i}Δα_{i}x_{i} + y_{j}Δα_{j}x_{j} = w^{old} + Δw  (14.79)

where Δw = w^{new} − w^{old} = y_{i}Δα_{i}x_{i} + y_{j}Δα_{j}x_{j}. To update b, consider:

ΔE_{k} = E_{k}^{new} − E_{k}^{old} = u_{k}^{new} − u_{k}^{old}
= ∑_{n=1}^{N} α_{n}^{new}y_{n}K_{nk} + b^{new} − ∑_{n=1}^{N} α_{n}^{old}y_{n}K_{nk} − b^{old}
= y_{i}Δα_{i}K_{ik} + y_{j}Δα_{j}K_{jk} + b^{new} − b^{old} (k = 1, 2) (14.80)

Consider the following cases:

• If 0 < α_{k} < C for either k = i or k = j, then y_{k}E_{k} = 0 according to Eq. (14.52). We let E_{k}^{new} = 0 and solve the equation above to get:

b_{k}^{new} = b_{k}^{old} − (E_{k}^{old} + y_{i}Δα_{i}K_{ik} + y_{j}Δα_{j}K_{jk}) (k = 1 or 2) (14.81)

• If 0 < α_{k} < C for both k = i and k = j, then b_{i}^{new} = b_{j}^{new}.

• If α_{k} = 0 or α_{k} = C for both k = i and k = 2, then E_{k} ≠ 0, and b_{i}^{new} ≠ b_{j}^{new}, their average b^{new} = (b_{i}^{new} + b_{j}^{new})/2 can be used.

The computation above for maximizing L(α_{i}, α_{j}) is carried out iteratively each time when a pair of variables α_{i} and α_{j} is selected to be updated. Specifically, we select a variable α_{i} that violates any of the KKT conditions given in Eq. (14.52) in the outer loop of the SMO algorithm, and then pick a second variable α_{j} in the inner loop of the algorithm, both to be optimized. The selection of these variables can be either random (e.g., by following their order), or guided by some heuristics, such as always choosing the variable with maximum step size Δα_{n} = α_{n}^{new} −α_{n}^{old}. This iterative process is repeated until convergence when the KKT conditions are satisfied by all α_{1}, ⋯, α_{N} variables, i.e., no more variables need to be updated. All data points corresponding to α_{n} ≠ 0 are the support vectors, based on which we can find the offset b by Eq. (14.55) and classify any unlabeled point x by Eq. (14.56).

The Matlab code for the essential part of the SMO algorithm is listed below, based mostly on a simplified SMO algorithm with some modifications.

In particular, the if-statement

if (alpha(i)<C & yE<-tol) | (alpha(i)>0 & yE>tol)

checks the three KKT conditions in Eq. (14.52), where yE = y_{n}E_{n} and tol is a small constant. The first half checks if the first two of the three conditions are violated, while the second half checks if the second two of the three conditions are violated.

[X y]=getData;          % get training data
[m n]=size(X);          % size of data
alpha=zeros(1,n);       % alpha variables
bias=0;                 % initial bias
it=0;                   % iteration index

while (it<maxit)        % number of iterations less than maximum
it=it+1;
changed_alphas=0;   % number of changed alphas
N=length(y);        % number of support vectors
for i=1:N           % for each alpha_i
Ei=sum(alpha.*y.*K(X,X(:,i)))+bias-y(i);
yE=Ei*y(i);
if (alpha(i)<C & yE<-tol) | (alpha(i)>0 & yE>tol)
         % KKT violation
for j=[1:i-1,i+1:N]   % for each alpha_j unequal alpha_i
Ej=sum(alpha.*y.*K(X,X(:,j)))+bias-y(j);
ai=alpha(i);      % alpha_i old
aj=alpha(j);      % alpha_j old
if y(i)==y(j)     % s=y_i y_j=1
    L=max(0,alpha(i)+alpha(j)-C);
    H=min(C,alpha(i)+alpha(j));
else              % s=y_i y_j=-1
    L=max(0,alpha(j)-alpha(i));
    H=min(C,C+alpha(j)-alpha(i));
end
if L==H           % skip when L==H
    continue
end
eta=2*K(X(:,j),X(:,i))-K(X(:,i),X(:,i))-K(X(:,j),X(:,j));
alpha(j)=alpha(j)+y(j)*(Ej-Ei)/eta;   % update alpha_j
if alpha(j) > H
    alpha(j) = H;
elseif alpha(j) < L
    alpha(j) = L;
end
if norm(alpha(j)-aj) < tol   % skip if no change
    continue
end
alpha(i)=alpha(i)-y(i)*y(j)*(alpha(j)-aj);
                                      % find alpha_i
bi = bias - Ei - y(i)*(alpha(i)-ai)*K(X(:,i),X(:,i))...
    -y(j)*(alpha(j)-aj)*K(X(:,j),X(:,i));
bj = bias - Ej - y(i)*(alpha(i)-ai)*K(X(:,i),X(:,j))...
    -y(j)*(alpha(j)-aj)*K(X(:,j),X(:,j));
if 0<alpha(i) & alpha(i)<C
    bias=bi;
 elseif 0<alpha(j) & alpha(j)<C
    bias=bj;
else

                bias=(bi+bj)/2;
            end
            changed_alphas=changed_alphas+1;  % one more alpha changed
        end
    end
end
if changed_alphas==0      % no more changed alpha, quit
    break
end
I=find(alpha~=0);         % indecies of non-zero alphas
alpha=alpha(I);           % find non-zero alphas
Xsv=X(:,I);               % find support vectors
ysv=y(I);                 % their corresponding indecies
end                           % end of iteration
nsv=length(ysv);              % number of support vectors

Here K(X,x) is a function that takes an m × n matrix X = [x_{1}, ⋯, x_{n}] and an n-D vector x and produces the kernel functions K(x_{i}, x), (i = 1, ⋯, n) as output.

function ker=K(X,x)
global kfunction
global gm                  % parameter for Gaussian kernel
[m n]=size(X);
ker=zeros(1,n);
if kfunction=='l'
    for i=1:n
        ker(i)=X(:,i).'*x; % linear kernel
    end
elseif kfunction=='g'
    for i=1:n
        ker(i)=exp(-gm*norm(X(:,i)-x)); % Gaussian kernel
    end
elseif kfunction=='p'
    for i=1:n
        ker(i)=(X(:,i).'*x).^3; % polynomial kernel
    end
end
end

Having found all non-zero α_{n} ≠ 0 and the corresponding support vectors, we further find the offset or bias term:

 bias=0;
 for i=1:nsv
    bias=bias+(ysv(i)-sum(ysv.*alpha.*K(Xsv,Xsv(:,i))));

Figure 14.7 Classification by SVM Based on SMO

    end
    bias=bias/nsv;

Any unlabeled data point x is classified into class C_{+} or C_{−}, depending on whether the following expression is greater or smaller than zero:

    y=sum(alpha.*ysv.*K(Xsv,x))+bias;

Example 14.3 The same training set used previously to test the SVM with a hard margin is used again to test the SVM with a soft margin. Two results are shown in Fig. 14.7, corresponding to two different values C = 1.2 and C = 0.05 for the weight of the error term ∑_{n=1}^{N} ξ_{n} in the objective function. We see that when C = 1.2 is large, the number of support vectors (the four solid dots) is small due to the small errors ξ_{n} allowed, and the decision boundary determined by the these support vectors independent of the rest of the data set (circles) is mostly dictated by the local points close to the boundary, not necessarily a good reflection of the global distribution of the data points. This result is similar to the previous one generated by the hard-margin SVM.

On the other hand, when C = 0.1 is small, a greater number of data points become support vectors due to the larger (the 13 solid dots) error ξ_{n} allowed, and the resulting linear decision line determined by these support vectors reflects global distribution of the data points, and it better separates the two classes in terms of their mean positions.

Example 14.4 The classification results for the XOR data set are shown in Fig. 14.8 with C = 1.2 and C = 0.05. As the two classes in the XOR data set are not linearly separable, a Gaussian or radial basis function (RBF) kernel is used to map the data from 2-D space to an infinite dimensional space, which is partitioned into two regions for the two classes. When C = 1.2 is large, the decision boundary is dictated by a relatively small number of support vectors (solid dots, with small error ξ_{n}) and all data points are classified correctly, but it is more prone to the overfitting problem, if some of the small number of support vectors are outliers due to noise.

Figure 14.8 Classification of XOR Dataset by SVM Based on SMO

Figure 14.9 Example of Classification by SMO

One the other hand, when C = 0.05 is small, the decision boundary is determined by a greater number of support vectors, better reflecting the global distribution of the data points. However, as the local points close to the boundary are not relatively de-emphosized, some miss-classification occurs, some nine red dots are classified into the blue region incorrectly.

Two additional examples of binary classification based on SMO are shown in Figs. 14.9 and 14.10

14.5 Multiclass Classification

The SVM algorithm is inherently a binary classifier, but it can be generalized to multiclass problems as well. In this section, we consider a general K-class classifier based on a training set X = [x_{1}, ⋯, x_{N}], of which each sample x_{n} is labeled by y_{n} ∈ {1, ⋯, K} to indicate x_{n} ∈ C_{y_{n}}.

We first consider two straight forward and imperial methods for multiclass classification based directly on binary SVM.

Figure 14.10 Example of Classification by SMO

• One-Vs-One (1V1)

Any unlabeled x is classified to a class which receives the maximum votes out of the K(K − 1)/2 binary classifications between every pair of the K classes.

• One-Vs-Rest (1VR)

This method converts a K-class problem (K > 2) into K binary problems. First, we regroup the K classes C_{1}, ⋯, C_{K} into two classes C_{+} = C_{k} and C_{−} = {C_{l}|l = 1, ⋯, K, l ≠ k} and find the corresponding decision function f_{k}(x) representing quantitatively to what extent a given x belongs to class C_{+} = C_{k} (if f_{k}(x) > 0), instead of C_{−} containing all remaining K − 1 classes (if f_{k}(x) < 0). After this process has been carried out for all k = 1, ⋯, K, the x can be classified as below:

if f_{l}(x) = max{f_{1}(x), ⋯, f_{K}(x)}, then x ∈ C_{l}  (14.82)

We now consider another method for multiclass classification by directly generalizing the binary SVM so that all K classes can be separated by the decision boundaries is such a way that the margin between any two of the K classes is maximized. First, as in the binary case, we define a linear function for each of the K classes:

f_{k}(x) = w_{k}^{T} x + b_{k} (k = 1, ⋯, K)  (14.83)

Once the parameters w_{k} and b_{k} are determined during training, any unlabeled point x can be classified as below:

if f_{l}(x) = max_{k}{f_{k}(x) (k = 1, ⋯, K)}, then x ∈ C_{l}  (14.84)

Fig. 14.11 illustrates the classification of K = 3 classes in d = 2 dimensional feature space. Here each straight line H_{k} (plane or hyperplane if d ≥ 3) is determined by equation f_{k}(x) = w_{k}^{T} x + b_{k} = 0, (k = 1, ⋯, K), with normal direction in w_{k} and distance d(H_{k}, 0) = |b_{k}|/||w_{k}|| to the origin (the same as in Eq. (14.4)).

Figure 14.11 Multiclass SMV

According to the classification rule in Eq. (14.84), the 2-D feature space is partitioned into three regions each for one of the K = 3 classes by the decision boundaries B_{12}, B_{23}, B_{31}, where B_{ij} between classes C_{i} and C_{j} is composed of all points x ∈ B_{ij} satisfying f_{i}(x) = f_{j}(x). Like in binary SVM case, we further define for each of the K classes two additional straight lines (planes or hyperplanes if d ≥ 3) in parallel with and on either side of H_{k}, denoted by H_{k±} and represented by equations f_{k}(x) = w_{k}^{T} x + b_{k} ± c_{k}. This distance between these straight lines and H_{k} is (the same as in Eq. (14.9)):

d(H_{k±}, H_{k}) = \frac{|c_{k}|}{||w_{k}||}  (14.85)

Also as shown in the figure, the intersections of these parallel lines (two for each of class) further determine two straight lines parallel to the decision boundary B_{12}, defining the decision margin between the support vectors of C_{1} and C_{2}. It can be seen that this decision margin is related to both |c_{1}|/||w_{1}|| and |c_{2}|/||w_{2}||, which are maximized if ||w_{1}|| and ||w_{2}|| are minimized.

Now the multiclass classification problem can be formulated as to find the parameters w_{k} and b_{k} for each of the K classes, so that ∑_{k=1}^{K} ||w_{k}||^{2} is minimized and thereby the decision margins for all boundaries B_{ij} are maximized, under the constraint that every training sample x_{n} labeled by y_{n} is correctly classified with a decision margin of 1 (the distance between the support vectors on opposite sides of B_{ij} is 2):

minimize: \frac{1}{2} ∑_{k=1}^{K} ||w_{k}||^{2}
subject to: w_{y_{n}}^{T}x_{n} + b_{y_{n}} ≥ w_{k}^{T} x_{n} + b_{k} + 2 (k = 1, ⋯, K, k ≠ y_{n}) (14.86)

Similar to Eq. (14.37) in soft margin SVM, the constraints of this optimization problem can be relaxed by including in each constraint an extra error term ξ_{nk} ≥ 0, which also needs to be minimized. Now the problem can be reformulated as

minimize: \frac{1}{2} ∑_{k=1}^{K} ||w_{k}||^{2} + C ∑_{k=1}^{K} ∑_{n=1}^{N} ξ_{nk}
subject to: w_{y_{n}}^{T}x_{n} + b_{y_{n}} ≥ w_{k}^{T} x_{n} + b_{k} + 2 − ξ_{nk}
ξ_{nk} ≥ 0
(n = 1, ⋯, N, k = 1, ⋯, K, k ≠ y_{n})  (14.87)

Once this constrained optimization problem is solved we get w_{k} and b_{k} for all k = 1, ⋯, K, and we can find f_{k}(x) = w_{k}^{T} x + b_{k} for any unlabeled sample x and classify it based on Eq. (14.84).

Similar to the binary SVM, we approach this optimization problem by considering the dual problem. We first construct the Lagrangian, the primal function, of this constrained optimization problem:

L_{p}(W, b, ξ, α, β) = \frac{1}{2} ∑_{k} ||w_{k}||^{2} + C ∑_{k} ∑_{n} ξ_{nk}

− ∑_{k} ∑_{n} α_{nk} [(w_{y_{n}} − w_{k})^{T} x_{n} + b_{y_{n}} − b_{k} − 2 + ξ_{nk}]
− ∑_{k} ∑_{n} β_{nk}ξ_{nk}  (14.88)

where W, b, ξ, α, and β are matrices and vectors containing the corresponding parameters w_{k}, b_{k}, ξ_{nk}, α_{nk} and β_{nk} for all k = 1, ⋯, K and n = 1, ⋯, N. For the special case k = y_{n} not included in the constraints, α_{ny_{n}} = β_{ny_{n}} = 0, and the constraint becomes an equality 0 = 2 − ξ_{ny_{n}}, i.e., ξ_{ny_{n}} = 2.

The KKT conditions for the Lagrangian in Eq. (14.88) are listed below (similar to Eqs. (14.40) through (14.45) for the Lagrangian in Eq. (14.39)):

• Stationarity:

–

\frac{∂}{∂w_{k}} L_{p}(w, b, ξ, α, β) = w_{k} + ∑_{n} α_{nk}x_{n} − ∑_{n} c_{nk}A_{n}x_{n} = 0 (14.89)

i.e.,

w_{k} = ∑_{n}(c_{nk}A_{n} − α_{nk})x_{n}  (14.90)

where we have defined

c_{nk} = \{ \begin{matrix}1 & if k = y_{n} \\ 0 & else \end{matrix} , and A_{n} = ∑_{k} α_{nk}  (14.91)

–

\frac{∂}{∂b_{k}} L_{p}(w, b, ξ, α, β) = − ∑_{n} c_{nk}A_{n} + ∑_{n} α_{nk} = 0  (14.92)

–

\frac{∂}{∂ξ_{nk}} L_{p}(w, b, ξ, α, β) = C − α_{nk} − β_{nk} = 0  (14.93)

• Primal and dual feasibilities:

(w_{y_{n}} − w_{k})^{T} x_{n} + b_{y_{n}} − b_{k} − 2 + ξ_{nk} ≥ 0
α_{nk} ≥ 0, β_{nk} ≥ 0, ξ_{nk} ≥ 0
(n = 1, ⋯, N, k = 1, ⋯, K, k ≠ y_{n})  (14.94)

• Complementarity:

α_{nk}[(w_{y_{n}} − w_{k})^{T} x_{n} + b_{y_{n}} − b_{k} − 2 + ξ_{nk}] = 0  (14.95)

These KKT conditions need to be satisfied by the solution of the optimization problem, including w_{k}^{∗}, b_{k}^{∗}, and ξ_{nk}^{∗} that need to satisfy the stationarity of the KKT conditions. We can therefore substitute Eq. (14.90) into Eq. (14.88) and get

L_{p}(W, b, ξ, α, β) = \frac{1}{2} ∑_{k} (∑_{m}(c_{mk}A_{m} − α_{mk})) (∑_{n}(c_{nk}A_{n} − α_{nk})) x_{m}^{T}x_{n}

− ∑_{k} ∑_{n} α_{nk} [∑_{m}(c_{my_{n}}A_{m} − α_{my_{n}})x_{m}^{T}x_{n} − ∑_{m}(c_{mk}A_{m} − α_{mk})x_{m}^{T}x_{n} + b_{y_{n}} − b_{k} − 2]

− ∑_{k} ∑_{n} α_{nk}ξ_{nk} + C ∑_{k} ∑_{n} ξ_{nk} − ∑_{k} ∑_{n} β_{nk}ξ_{nk}  (14.96)

This expression can be significantly simplified. First, we note that the last three terms cancel out due to Eq. (14.93). Second, the two terms containing b_{y_{n}} and b_{k} can be written as

∑_{k} ∑_{n} α_{nk}b_{y_{n}} = ∑_{k} ∑_{n} c_{nk}α_{nk}b_{k} = ∑_{k} b_{k} ∑_{n} A_{n}c_{nk}
∑_{k} ∑_{n} α_{nk}b_{k} = ∑_{k} b_{k} ∑_{n} α_{nk}  (14.97)

and they also cancel out due to Eq. (14.92).

Now we can rewrite L_{p} as a function of only the variable α:

L_{d}(α) =^{1} 2 ∑_{k} ∑_{n} α_{nk}

+ ∑_{k} ∑_{m} ∑_{n} [\frac{1}{2}c_{mk}c_{nk}A_{m}A_{n} − c_{mk}α_{nk}A_{m} + \frac{1}{2}α_{mk}α_{nk} − c_{my_{n}}α_{nk}A_{m} + α_{nk}α_{my_{n}} + c_{mk}α_{nk}A_{m} − α_{nk}α_{mk}] x_{m}^{T}x_{n}
= 2 ∑_{k} ∑_{n} α_{nk} + ∑_{k} ∑_{m} ∑_{n} [\frac{1}{2}c_{mk}c_{nk}A_{m}A_{n} − \frac{1}{2}α_{mk}α_{nk} − c_{my_{n}}α_{nk}A_{m} + α_{nk}α_{my_{n}}] x_{m}^{T}x_{n}
=^{2} 2 ∑_{k} ∑_{n} α_{nk} + ∑_{k} ∑_{m} ∑_{n} [−\frac{1}{2}c_{my_{n}}A_{m}A_{n} − \frac{1}{2}α_{mk}α_{nk} + α_{nk}α_{my_{n}}] x_{m}^{T}x_{n}  (14.98)

where the labeled equal signs are due to the following:

1. The two cross terms of the second part can be combined as:

∑_{m} ∑_{n} ∑_{k} c_{mk}α_{nk}A_{m} = ∑_{m} ∑_{n} ∑_{k} c_{nk}α_{mk}A_{n}  (14.99)

2. As c_{my_{n}} = 1 if y_{m} = y_{n} but zero otherwise, we have

∑_{k} c_{mk}c_{nk} = c_{my_{n}} = \{ \begin{matrix}1 & y_{m} = y_{n} \\ 0 & otherwise \end{matrix}  (14.100)

and the first term of the second part can be written as A_{m}A_{n}c_{my_{n}}/2. Also, due to Eq. (14.92), the third term can be written as −c_{my_{n}}c_{nk}A_{n}A_{m} = −c_{my_{n}}A_{n}A_{m} (when summed over k) and combined with the first one.

Eq. (14.98) is the quadratic dual function that needs to be maximized with respect to α for it to be the tightest lower bound, subject to the linear constraint imposed by Eq. (14.92). We therefore have the following dual problem:

maximize: L_{d}(α)
subject to: ∑_{n} A_{n}c_{nk} = ∑_{n} α_{nk}
0 ≤ α_{nk} ≤ C, α_{ny_{n}} = 0
(n = 1, ⋯, N, k = 1, ⋯, K, k ≠ y_{n})  (14.101)

This is a quadratic programming (QP) problem containing NK variables α_{nk}, (n = 1, ⋯, k = 1, ⋯, K), solving which we get the optimal α^{∗}.

We can further get all K optimal weight vectors w_{k}^{∗} by Eq. (14.90), and b^{∗} based on the complementarity of the KKT conditions, by solving the simultaneous equations in Eq. (14.95) (similar to how we get b^{∗} in binary case in Eq. (14.30). Having found both optimal parameters w_{k}^{∗} and b_{k}^{∗}, we can classify any unlabeled pattern x by Eq. (14.84).

We note that ass the data points in Eq. (14.98) appear only in the form of an inner product, kernel method can be used to map the problem into a high dimensional space where the data points are better separable.

The algorithm discussed here can be considered the most classic multiclass SVM algorithm as it is the first SVM algorithm to address multiclass problems by directly generalizing the binary SVM. Due to some of its drawback such as the size of variable α in the dual problem (NK versus N in Eq. (14.27) in the binary case), various algorithms are developed later for improvement in computational complexity.

14.6 Kernelized Bayes classifier

The naive Bayes classification is based on a quadratic decision function and is therefore unable to classify data that are not quadratically separable. However, as shown below, the Bayes method can be reformulated so that all data points appear in the form of an inner product, and the kernel method can be used to map the original space into a higher dimensional space in which all groups can be separated even though they are not quadratically separable in the original space.

Consider a binary Bayes classifier by which any sample x is classified into either of the two classes C_{+} and C_{−} depending on whether x is on the positive or negative side of the quadratic decision surface (Eq. (13.23)):

f(x) = x^{T} Wx + w^{T} x + w \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix}  (14.102)

As show in Eq. (13.22), here

W = −\frac{1}{2}(Σ_{+}^{−1} − Σ_{−}^{−1})
w = Σ_{+}^{−1}m_{+} − Σ_{−}^{−1}m_{−}
w = −\frac{1}{2}(m_{+}^{T}Σ_{+}^{−1}m_{+} − m_{−}^{T}Σ_{−}^{−1}m_{−}) − \frac{1}{2} log \frac{|Σ_{+}|}{|Σ_{−}|} + log \frac{P (C_{+})}{P (C_{−})}(14.103)

where Σ_{+} and Σ_{−} are respectively the covariance matrices of the samples in C_{+} and C_{−},

m_{+} = \frac{1}{N_{+}} ∑_{x∈C_{+}} x, m_{−} = \frac{1}{N_{−}} ∑_{x∈C_{−}} x  (14.104)

are their mean vectors, and P_{+} = N_{+}/N and P_{−} = N_{−}/N are their prior probabilities. Specially if Σ_{+} = Σ_{−} = Σ and therefore W = 0, the quadratic decision surface becomes a linear decision plane described by

f(x) = w^{T} x + w = [Σ^{−1}(m_{+} − m_{−})]^{T} x + w = 0  (14.105)

in the same form as the decision equation for the support vector machine. An unlabeled sample x is classified into either of the two classes depending on on which side of a threshold −w its projection onto the normal direction w of the decision plane lies.

As matrices W, Σ_{+}^{−1}, and Σ_{−}^{−1} are all symmetric, they can be written in the following eigen-decomposition form:

W = VΛV^{T} = UU^{T} , Σ_{+}^{−1} = V_{+}Λ_{+}V_{+}^{T} = U_{+}U_{+}^{T}, Σ_{−}^{−1} = V_{−}Λ_{−}V_{−}^{T} = U_{−}U_{−}^{T}

(14.106)

where U = VΛ^{1/2}, U_{+} = V_{+}Λ_{+}^{1/2} and U_{−} = V_{−}Λ_{−}^{1/2}. We can write vector w as:

w = Σ_{+}^{−1}m_{+} − Σ_{−}^{−1}m_{−} = U_{+}U_{+}^{T} \frac{1}{N_{+}} ∑_{x_{+}∈C_{+}} x_{+} − U_{−}U_{−}^{T} \frac{1}{N_{−}} ∑_{x_{−}∈C_{−}} x_{−}

(14.107)

Any unlabeled sample x can now be classified into either of the two classes based on its decision function:

f(x) = x^{T} Wx + w^{T} x + w
= x^{T} UU^{T} x + (\frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}U_{+}U_{+}^{T}) x − (\frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}U_{−}U_{−}^{T}) x + w

= z^{T} z + \frac{1}{N_{+}} ∑_{z_{++}} (z_{++}^{T}z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} (z_{−−}^{T}z_{−}) + w  (14.108)

where

\{ \begin{matrix}z_{++} = U_{+}^{T}x_{+} (x_{+} ∈ C_{+}) \\ z_{−−} = U_{−}^{T}x_{−} (x_{−} ∈ C_{−}) \end{matrix} , \{ \begin{matrix}z = U^{T} x \\ z_{+} = U_{+}^{T}x \\ z_{−} = U_{−}^{T}x \end{matrix}  (14.109)

As all data points appear in the form of an inner product, the kernel method can be used by replacing all inner products in the decision function by the corresponding kernel functions:

f(x) = K(z, z) + \frac{1}{N_{+}} ∑_{z_{++}} K(z_{++}, z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} K(z_{−−}, z_{−}) + b = p(x) + b

(14.110)

This is the decision function in the new higher dimensional space, where p(x) is defined as a function composed of all terms in f(x) except the last offset term b, which is to replace w in the original space. To find this b, we first map all training samples into a 1-D space x_{n} = p(x_{n}), (n = 1, ⋯, N) and sort them together with their corresponding labelings y_{1}, ⋯, y_{N}, and then search through all N −1 possible ways to partition them into two groups indexed respectively by 1, ⋯, k and k + 1, ⋯, N to find the optimal k corresponding to the maximum labeling consistency measured by

| ∑_{n=1}^{k} y_{n}| + | ∑_{n=k+1}^{N} y_{n}|, (k = 1, ⋯, N − 1)  (14.111)

The middle point (x_{k} + x_{k+1})/2 between x_{k} and x_{k+1} is used as the optimal threshold to separate the training samples into two classes in the 1-D space, i.e., the offset is b = −(x_{k} + x_{k+1})/2. Now the unlabeled point x can be classified into either of the two classes C_{+} and C_{−}:

p(x) + b \{ \begin{matrix}> 0, & x ∈ C_{+} \\ < 0, & x ∈ C_{−} \end{matrix}  (14.112)

In general, when all data points are kernel mapped to a higher dimensional space, the two classes can be more easily separated, even by a hyperplane based on the linear part of the decision function without the second order term. This allows the assumption that the two classes have the same covariance matrix so that W = −(Σ_{+} − Σ_{−})/2 = 0 and the second order term is dropped. This is the justification for the following two special cases:

• If we approximate both Σ_{+} and Σ_{−} by their average Σ = (Σ_{+} +Σ_{−})/2, then W = 0 and the decision function of any x becomes

f(x) = w^{T} x + w = [Σ^{−1}(m_{+} − m_{−})]^{T} x + w
= [UU^{T} (\frac{1}{N_{+}} ∑_{x_{+}} x_{+} − \frac{1}{N_{−}} ∑_{x_{−}} x_{−})]^{T} x + w

= \frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}UU^{T} x − \frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}UU^{T} x + w
= \frac{1}{N_{+}} ∑_{z_{+}} z_{+}^{T}z − \frac{1}{N_{−}} ∑_{z_{−}} z_{−}^{T}z + w  (14.113)

where UU^{T} = Σ^{−1}, z_{+} = U^{T} x_{+}, z_{−} = U^{T} x_{−}, and z = U^{T} x. This decision function can be converted to the following if the kernel method is used:

f(x) = \frac{1}{N_{+}} ∑_{z_{+}} K(z_{+}, z) − \frac{1}{N_{−}} ∑_{z_{−}} K(z_{−}, z) + b  (14.114)

• More specially, if Σ_{+} = Σ_{−} = I, then the decision function above becomes

f(x) = w^{T} x + w = (m_{+} − m_{−})^{T} x + w
= \frac{1}{N_{+}} ∑_{x_{+}} x_{+}^{T}x − \frac{1}{N_{−}} ∑_{x_{−}} x_{−}^{T}x + w  (14.115)

which can be converted to the following if the kernel method is used:

f(x) = \frac{1}{N_{+}} ∑_{x_{+}} K(x_{+}, x) − \frac{1}{N_{−}} ∑_{x_{−}} K(x_{−}, x) + b  (14.116)

Note that if the kernel method is used to replace an inner product by a kernel function K(x, x′) = ϕ(x)^{T} ϕ(x′), we need to map all data points to ϕ(x_{n}), (n = 1, ⋯, N) in the higher dimensional space, instead of only mapping their means ϕ(m_{+}) and ϕ(m_{−}), because the mapping of a sum is not equal to the sum of the mapped points if the kernel is not linear:

ϕ(m_{+}−m_{−}) ≠ ϕ(m_{+})−ϕ(m_{−}), ϕ(m_{±}) = ϕ (\frac{1}{n_{±}} ∑_{x∈C_{±}} x) ≠ \frac{1}{n_{±}} ∑_{x∈C_{±}} ϕ(x)

(14.117)

The three cases above are summarized below:

• Type I, Eq. (14.116)

f(x) = \frac{1}{N_{+}} ∑_{x_{+}} K(x_{+}, x) − \frac{1}{N_{−}} ∑_{x_{−}} K(x_{−}, x) + b  (14.118)

• Type II, Eq. (14.113)

f(x) = \frac{1}{N_{+}} ∑_{z_{+}} K(z_{+}, z) − \frac{1}{N_{−}} ∑_{z_{−}} K(z_{−}, z) + b  (14.119)

• Type III, Eq. (14.110)

f(x) = z^{T} z + \frac{1}{N_{+}} ∑_{z_{++}} K(z_{++}, z_{+}) − \frac{1}{N_{−}} ∑_{z_{−−}} K(z_{−−}, z_{−}) + b (14.120)

The Matlab code for the essential part of the algorithm is listed below. Given X and y for the data array composed of N training vectors X = [x_{1}, ⋯, x_{N}] each labeled by the corresponding component y = [y_{1}, ⋯, y_{N}], the code carries out the training and then classifies any unlabeled data point into either of the two classes. Parameter type selects any one of the three different versions of the algorithm, and the function K(X,x) returns a 1-D array containing all N kernel functions K(x_{i}, x), (i = 1, ⋯, N) of the column vectors in X and vector x.

    X=getData;
    [m n]=size(X);
    X0=X(:,find(y>0));   n0=size(X0,2); % separate C+ and C-
    X1=X(:,find(y<0));   n1=size(X1,2);
    m0=mean(X0,2);    C0=cov(X0');      % mean and covariance
    m1=mean(X1,2);    C1=cov(X1');
    if type==1
        for i=1:n
            x(i)=sum(K(X0,X(:,i)))/n0-sum(K(X1,X(:,i)))/n1;
        end
    elseif type==2
        C=inv(C0+C1);   [V D]=eig(C);   U=(V*D^(1/2))';
        Z=U*X;  Z0=U*X0;   Z1=U*X1;
        for i=1:n
            x(i)=sum(K(Z0,Z(:,i)))/n0-sum(K(Z1,Z(:,i)))/n1;
        end

elseif type==3
C0=inv(C0);   C1=inv(C1);   W=-(C0-C1)/2;
[V D]=eig(W);     U=(V*D^(1/2)).';    Z=U*X;
[V0 D0]=eig(C0);  U0=(V0*D0^(1/2))';  Z0=U0*X;  Z00=U0*X0;
[V1 D1]=eig(C1);  U1=(V1*D1^(1/2))';  Z1=U1*X;  Z11=U1*X1;
for i=1:n
    x(i)=K(Z(:,i),Z(:,i))+sum(K(Z00,Z0(:,i)))/n0-sum(K(Z11,Z1(:,i)))/n1;
end
end
[x I]=sort(x);   y=y(I);    % sort 1-D data and their labelings
smax=0;
for i=1:n-1                 % find optimal threshold value b
s=abs(sum(y(1:i)))+abs(sum(y(i+1:n)));
if s>smax
    smax=s;  b=-(x(i)+x(i+1))/2;
end
end

Note that W = −(Σ_{0}^{−1} − Σ_{1}^{−1})/2 may be either positive or negative definite, and its eigenvalue matrix D may contain negative values and D^{1/2} may contain complex values. Given any unlabeled data point x, the code below is carried out

if type==1
y(i)=sum(K(X0,x))/n0-sum(K(X1,x))/n1+b;
elseif type==2
Z=U*x;
y(i)=sum(K(Z0,z))/n0-sum(K(Z1,z))/n1+b;
elseif type==3
z=U*x;    z0=U0*x;   z1=U1*x;
y(i)=K(z,z)+sum(K(Z00,z0))/n0-sum(K(Z11,z1))/n1+b;
end

to classify x into either C_{+} if y > 0, or C_{−} if y < 0.

In comparison with the SVM method, which requires solving a QP problem by certain iterative algorithm (either the interior point method or the SMO method), the kernel based Bayes method is closed-form and therefore extremely efficient computationally. Moreover, as shown in the examples below, this method is also highly effective as its classification results are comparable and often more accurate than those of the SVM method.

We now show a few examples to test all three types of the kernel based Bayes method based on a set of simulated 2-D data. Both linear kernel and RBF kernel K(x, y) = exp(−γ||x − y||). The value of the parameter γ used in the examples is 5, but it can be fine tuned in a wide range (e.g., 1 < γ < 9 to make proper tradeoff between accuracy and avoiding overfitting. The performances of these

Figure 14.12 Example 1 of Kernelized Bayes Classifier

method are also compared with the SVM method implemented by the Matlab function:

fitcsvm(X',y,'Standardize',true,'KernelFunction','linear','KernelScale','auto'))

Example 14.5 Four different binary classifiers are applied to two 2-D datasets, with their results compared in terms of the correct rates isted below, and their partitionings of the 2-D space shown in Figs. 14.12 and 14.13.

Kernel

Matlab SVM Kernel Bayes I Kernel Bayes II

Kernel Bayes III

Test 1 linear  93.0%  88.0%  94.0%  94.0%
Test 2 linear  73.0%  80.0%  79.5%  96.5%
Test 3 RBF  98.5%  100%  100%  100%

(14.121)

• Test 1: Two sets of data are generated based on the following mean vectors and covariance matrices:

m_{+} = [ \begin{matrix}−1 \\ 0 \end{matrix} ] , m_{−} = [ \begin{matrix}+1 \\ 0 \end{matrix} ] , Σ_{+} = Σ_{−} = 3 × [ \begin{matrix}1 & 0.5 \\ 0.5 & 0.5 \end{matrix} ]

(14.122)

The linear kernel is used for all methods. The kernel Bayes methods II and III achieve the best result (94%), slightly better than that of the standard SVM (93%). But the kernel Bayes method I performs poorly (84.7%), as it is based only on the means of the two classes without taking into consideration their covariances representing the distribution of the data points.

• Test 2: Two sets of data not linearly separable are classified by the four methods all based on the linear kernel. The kernel Bayes III achieves the best result (96.5%), due to its quadratic term in the decision function, the kernel Bayes I and II perform much more poorly, but still slightly better than the SVM method.

• Test 3: The same dataset used in Test 2 is classified but now based on the RBF kernel. While the SVM performs performs reasonably well (98.5%), all three versions of the kernel Bayes method (with γ = 5) achieve the perfect result with all points of the two classes completely separated. However, the partitioning of the space by the kernel Bayes III is highly fragmented, indicating an obvious overfitting.

Figure 14.13 Example 2 of Kernelized Bayes Classifier

Figure 14.14 Example 3 of Kernelized Bayes Classifier

Example 14.6 Three datasets (XOR, cocentric rings, multi-clusters) are gener-

ated using the Matlab code at: https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html.

The following four tests are carried out for the three versions of the kernel Bayes all based on a parameter γ = 5.

• Test 1: Exclusive OR dataset, linear kernel

As the data are not linearly separable, all methods performed poorly except kernelized Bayes Type III with a second order term in the decision function. The results are shown in Fig. 14.15.

• Test 2: Exclusive OR dataset, RBF kernel

All four methods performed very well, but all three variations of the kernelized Bayes method achieved higher correct rates than the SVM. The results are shown in Fig. 14.16.

• Test 3: Cocentric ring dataset, RBF kernel. The results are shown in Fig.

14.17.

• Test 4: Multi-cluster dataset, RBF kernel. The results are shown in Fig. 14.18.

The classification correct rates of the four tests are listed below:

Kernel

Matlab SVM Kernel Bayes I Kernel Bayes II

Kernel Bayes III

Test 1 Linear  58.75%  60.0%  60.25%  97.0%
Test 2 RBF  97.75%  98.50%  98.50%  99.50%
Test 3 RBF  98.0%  100%  100%  100%
Test 4 RBF  96.0%  98.5%  95.5%  97.0%

Figs. through 14.18.

Example 14.7 In this example both the kernel Bayes and SVM methods (by

Matlab function fitcsvm, see https://www.mathworks.com/help/stats/fitcsvm.html)

are applied to the iris dataset. Their results are compared in Fig. 14.19. Here

Figure 14.15 Example 14.6 test 1

Figure 14.16 Example 14.6 test 2

only two (3rd and 4th) of the four features are used, with half of the samples used for training while the other half for testing. Note that the second class (setosa in green) and third class (versicolor in blue) not linearly separable can be better separated by the kernel Bayes method.

Problems

In this homework, you are to develop your own code to implement the binary and multiclass SVM algorithms. You are also encouraged to compare your results with those obtained by the built-in SVM functions in Matlab.

1. Develop code to implement the SVM algorithm using your own function for quadratic programming (QP) by the interior point algorithm (Section 3.7), and built-in QP function quadprog in Matlab, based on the radial basis function (RBF) kernel K(x_{i}, x_{j}) = c exp(−γ||x_{i} − x_{j}||^{2}).

Apply your code to carry out binary classification for each of the following 2-D datasets. Experiment with different values for the two hyper-parameters c

Figure 14.17 Example 14.6 test 3 Figure 14.18 Example 14.6 test 4

Figure 14.19 Comparison of kernel Bayes and SVM applied to iris data

and γ in the kernel function and check to see how they affect the classification result.

Visualize the partitioning of the 2-D feature space by classifying all points in the space (with proper resolution) into either of the two classes, same as the examples in

• /e176/programs/data60.txt

• /e176/programs/data61.txt

• /e176/programs/data62.txt

• /e176/programs/data63.txt

• /e176/programs/data64.txt

2. Carry out multiclass classification for both the iris and handwritten datasets, using your binary SVM code and the method of 1V1 or 1VR. Show your classification results in the form of confusion matrix as well as error rates.

3. Develop code for multiclass SVM and carry out classification for the iris and handwritten datasets. Show the classification results in the form of confusion matrix as well as error rates.

15 Clustering Analysis

All supervised classification algorithms previously considered are based on the assumed availability of some training data in which each of the data patterns x_{n} ∈ X = [x_{1}, ⋯, x_{N}}] is labeled by y_{n} ∈ y = [y_{1}, ⋯, y_{n}]^{T}, i.e., the class they each belong is known. However, when such training set is not available, there is still the need to explore some potential structure of the data in the form of a set of unknown number K of clusters {C_{1}, ⋯, C_{K}}, each composed of a set of similar data points close to each other in the feature space. This can be accomplished by the unsupervised method of clustering analysis, based only on the given dataset X = [x_{1}, ⋯, x_{N}], without any additional prior knowledge, such as the labeling of the data samples. We will now consider a few algorithms for clustering analysis.

15.1 K-means clustering

As suggested by the name of K-means clustering, in this method a set of K mean vectors m_{1}, ⋯, m_{K} are used to represent the clusters in the feature space, based on the assumption that there exist a set of K clusters in the dataset.

The K-means clustering algorithm can be formulated as an optimization problem to minimize an objective function

J = ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk}||x_{n} − m_{k}||^{2}  (15.1)

where

P_{nk} = \{ \begin{matrix}1 & if k = arg min_{l} ||x_{n} − m_{l}|| \\ 0 & otherwise \end{matrix}  (15.2)

indicates that x_{n} is assigned to the kth cluster C_{k} if its distance to the mean m_{k} is minimum among all K such distances. To minimize the objective function J with respective to m_{k}, we set its derivative with respect to each m_{k} to zero:

\frac{d}{dm_{k}} J = 2 ∑_{n=1}^{N} P_{nk}||x_{n} − m_{k}|| = 0 k = 1, ⋯, K  (15.3)

and solve the resulting equation for m_{k}:

m_{k} = \frac{∑_{n=1}^{N} P_{nk}x_{n}}{∑_{n=1}^{N} P_{nk}} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x (k = 1, ⋯, K)  (15.4)

where N_{k} = ∑_{n=1}^{N} P_{nk} is the number of all samples assigned to cluster C_{k}, and the summation is over all samples assigned to C_{k}.

As P_{nk} depends on m_{k} which in turn also depends on P_{nk}, the steps above can be carried out iteratively, starting with some K randomly initialized mean vectors, which are then modified iteratively until convergence, as shown in the following steps:

1. Set t = 0, and initialize randomly the K mean vectors, such as any K samples randomly chosen from the dataset: m_{1}^{(0)}, m_{2}^{(0)}, ...., m_{K}^{(0)}, set iteration index to zero t = 1;

2. Assign each sample x in the dataset to one of the K clusters according to its distance to the corresponding mean vector:

if ||x − m_{k}^{(t)}||^{2} = min_{1≤l≤K} ||x − m_{l}^{(t)}||^{2}, then x ∈ C_{k}^{(t)}  (15.5)

3. Update the mean vectors to get the new mean m_{k}^{(t+1)} so that the objective function J given above is minimized:

m_{k}^{(t+1)} = \frac{1}{N_{k}} ∑_{x∈C_{k}} x, (k = 1, ⋯, K)  (15.6)

4. Terminate if the algorithm has converged:

m_{k}^{(l+1)} = m_{k}^{(t)} (k = 1, ⋯, K)  (15.7)

Otherwise, t ← t + 1, go back to Step 2.

The K-means clustering is simple and effective, but it has the main drawback that the ground truth number of clusters in the dataset is typically unknown, and it may be difficult to determine its vallue without any prior knowledge. One way to address this issue is to carry out the clustering process multiple times with different K values, and then determine the optimal K value by comparing the clustering results to see which best reflects the intrinsic structure of the dataset in terms of how the data points are distributed. How well the clustering result fits the data can be measured quantitatively by the separability of the clusters, such as the ratio between the intra and inter-cluster distances, or the ration between-cluster scatteredness and the total scatteredness of the entire dataset, i.e., tr(S_{T}^{−1}S_{B}) in Eq. (9.24). The K value corresponding to the highest separability among all resulting clusters can be adopted as the optimal one. Such a quantitative measurement can also be used as a criterion other clustering algorithms, including the method of competitive learning network to be considered in Section 19.1.

The issue of unknown parameter K can also be addressed by modifying its value during the iteration. For example, if the inter-cluster distance between two small clusters is smaller than a predetermined threshold, they can be merged into one, also, if the intra-cluster distance of a large cluster is greater than a predetermined threshold, it can be split into two. This method of modifying the clustering results by merging and splitting leads to the algorithm of Iterative Self-Organizing Data Analysis Technique (ISODATA). However, this algorithm is highly empirical and it suffers from the drawback that it may be difficult to predetermine those thresholds without any prior knowledge or experience. It may therefore become an highly interactive process during which the threshold values themselves may need to be modified to better fit the data.

The Matlab code for the iteration loop of the K-means algorithm is listed below, where Mold and Mnew are respectively the K mean vectors before and after each modification. The iteration terminates when the mean vectors no longer change.

    Mnew=X(:,randi(N,1,K));     % pick any K samples as initial means
    er=inf;
    while er>0                  % main iteration
        Mold=Mnew;
        Mnew=zeros(d,K);        % initialize new means
        Nk=zeros(1,K);
        for n=1:N               % for all N samples
            x=X(:,n);
            dmin=inf;
            for k=1:K           % for all K clusters
                d=norm(x-Mold(:,k));
                if d<dmin
                    dmin=d;  j=k;
                end
            end
            Nk(j)=Nk(j)+1;
            Mnew(:,j)=Mnew(:,j)+x;
        end
        for k=1:K
            if Nk(k)>0
                Mnew(:,k)=Mnew(:,k)/Nk(k);
            end
        end
        er=norm(Mnew-Mold);     % terminate if means no longer change
    end

Example 15.1 The K-means algorithm is applied to a simulated dataset in 3-D space with C = 4 clusters. The three panels in Fig. 15.1 show the results corresponding to K = C − 1 (left), K = C (middle), and K = C + 1 (right). The

Figure 15.1 K-Means Clustering with K = 3 (left), K = 4 (middle), and K = 5 (right)

initial positions of the K means are marked by the black squares, while their subsequent positions through out the iteration are marked by smaller dots. The iteration terminates once the means have moved to the centers of the clusters and no longer change positions.

The clustering results corresponding to K = 3, 4, 5 can be evaluated by the separability tr(S_{T}^{−1}S_{B}), the K intra-cluster distance tr(Σ_{k}) (k = 1, ⋯, K) of the resulting clusters:

K = C − 1 = 3  K = C = 4  K = C + 1 = 5
Separability  1.76  2.56  2.58

Intra-cluster distance 9.1 44.3 11.8 10.8 12.7 11.1 9.1 10.8 9.1 11.1 8.9 9.4

(15.8)

and the K(K − 1)/2 inter-cluster (Bhattacharyya) distances (between any two of the K clusters):

\begin{matrix}K = C − 1 = 3 \\ 1 & 2 \\ 2 & 10.9 \\ 3 & 21.9 & 184.7 \end{matrix}  \begin{aligned}& K = C = 4 \\ 1 & 2 & 3 \\ 2 & 4.0 \\ 3 & 5.1 & 1.6 \\ 4 & 2.4 & 2.3 & 4.4 \end{aligned}  \begin{aligned}& K = C + 1 = 5 \\ 1 & 2 & 3 & 4 \\ 2 & 4.0 \\ 3 & 5.1 & 1.6 \\ 4 & 7.0 & 4.3 & 0.3 \\ 5 & 17.3 & 15.5 & 8.9 & 11.1 \end{aligned}

(15.9)

We see that when K = 3 < C = 4, the intra-cluster distance of the 2nd cluster is significantly greater than the other two, indicating the cluster may contain two smaller clusters. Also, when K = 5 > C = 4, the inter-cluster distance between clusters 3 and 4 is significantly smaller than others, indicating the two clusters are too close and can therefore be merged.

While the K-means method is simple and effective, it has the main shortcoming that the number of clusters K needs to be predetermined while it is typically unknown ahead of time without any prior knowledge. In this case, one could try different K values and compare the corresponding results in terms of the

Figure 15.2 K-Means Clustering with Merge and Split

intra and inter cluster distances, as well as the separabilities of the resulting clusters, as shown in the example above. Moreover, if the intra-cluster distance of a cluster is too large indicating it may contain more than one cluster (e.g., K = 3 in the example), it can be split; on the other hand, if the inter-cluster distance between two clusters is too small (e.g., K = 5 in the example), the two clusters may belong to the same cluster and need to be merged. Following such merging and/or splitting, a few more iterations can be carried out to make sure the final clustering results are optimal. Fig. 15.2 shows the clustering results of the same dataset above but modified by merging and splitting. The left panel is for K = 3, but the second cluster (green) is split, while the right panel is for K = 5, when the 4th (yellow) and 5th (cyan) clusters are merged.

Example 15.2 The K-means clustering method is applied to the dataset of handwritten digits from 0 to 9 used previously. The clustering result is visualized based on the KLT that maps the data samples in the original 256-D space into the 3-D space spanned by the first three eigenvectors with the greatest eigenvalues of the covariance matrix of the dataset. The ground truth labelings are color coded as shown on the left, while the clustering result is shown on the right. We see that the clustering results match the original data reasonably well.

The clustering result can also be shown in the confusion matrix previously used for supervised classification. Here the columns and rows represent the class labelings (the ground truth not used clustering) and the columns represent the clusters based on the clustering result, i.e., the element in the ith row and jth column is the number of samples labeled to belong to the ith class but assigned to the jth cluster. Ideally, there should be only one nonzero value in each row indicating all samples in a class are assigned to the same cluster, and one nonzero value in each column indicating all samples assigned to a cluster are from the same class. Based on this confusion matrix, we can also define the error rate as the ratio between the number of miss-clustered samples, the sum of all components in the confusion matrix excluding the greatest one in each row, and he total number of samples, sum of all components in the matrix.

Figure 15.3 K-Means Clustering of Handwritten Digits, ground truth classes (left) vs. clusters (right)

[ \begin{matrix}1 & 0 & 25 & 165 & 27 & 2 & 3 & 0 & 1 & 0 \\ 0 & 209 & 0 & 0 & 2 & 0 & 13 & 0 & 0 & 0 \\ 2 & 3 & 2 & 0 & 8 & 2 & 14 & 151 & 0 & 42 \\ 103 & 0 & 0 & 1 & 5 & 3 & 83 & 0 & 0 & 29 \\ 0 & 28 & 1 & 0 & 15 & 2 & 62 & 0 & 115 & 1 \\ 44 & 0 & 1 & 1 & 167 & 0 & 8 & 0 & 3 & 0 \\ 0 & 6 & 160 & 14 & 44 & 0 & 0 & 0 & 0 & 0 \\ 1 & 6 & 0 & 0 & 3 & 122 & 74 & 2 & 1 & 15 \\ 82 & 1 & 1 & 1 & 33 & 3 & 47 & 2 & 0 & 54 \\ 4 & 2 & 0 & 1 & 0 & 28 & 119 & 1 & 64 & 5 \end{matrix} ]  (15.10)

We note that as the resulting clusters are in random order unrelated to the ordering of the class identities in the dataset (ground truth not used in clustering), it is in general not the case (as in supervised classification) that the diagonal components of the confusion matrix always take larger values while all off-diagonal components take small values (ideally zero). In clustering, the largest value, assumed to be in the jth column, in each row, assumed to be the ith one, indicates that most of the samples known to belong to the ith class are assigned to the jth cluster. For a good clustering result, there should be only one large value in each row indicating most of the samples in a class are assigned to the same cluster, and there should be only one large value in each column indicating most samples assigned to a cluster are from the same class.

To further visualize the clustering result obtained by the K-means method, we convert all data points from 256-D vectors back into 16 × 16 image form, and then display the averages of all samples in each of the ten cluster as shown

Figure 15.4 Visualization of the Clusters Found by K-Means

in Fig. 15.4. We see that each of the ten clusters clearly represents one distinct digit.

15.2 Gaussian mixture model

The K-means clustering assigns each data point x to one of the K clusters based on the Euclidean distances between the point and the means of all clusters, without considering the overall distribution of the data points in each cluster represented by the covariance as well as the mean.

This issue can be addressed by the method of Gaussian mixture model (GMM) based on the assumption that each of the K clusters {C_{1}, ⋯, C_{K}} in the given dataset can be modeled by a Gaussian distribution N (m_{k}, Σ_{k}), (k = 1, ⋯, K) in terms of the mean vector m_{k} and covariance matrix Σ_{k}. The overall distribution of the entire dataset X = [x_{1}, ⋯, x_{N}] is the weighted sum of the K Gaussian distributions:

p(x) = ∑_{k=1}^{K} P_{k}N (x, m_{k}, Σ_{k})

= ∑_{k=1}^{K} P_{k} [\frac{1}{(2π)^{d/2}|Σ_{k}|^{1/2}} exp (−\frac{1}{2}(x − m_{k})^{T} Σ_{k}^{−1}(x − m_{k}))] (15.11)

where P_{k} is the weight for the kth Gaussian N (m_{k}, Σ_{k}), satisfying

∫_{−∞}^{∞} p(x) dx = ∑_{k=1}^{K} P_{k} ∫_{−∞}^{∞} N (x, m_{k}, Σ_{k}) dx = ∑_{k=1}^{K} P_{k} = 1  (15.12)

The GMM clustering is a method of expectation maximization (EM), by which all model parameters denoted by θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} is first estimated based on the given dataset, and then all data samples in the dataset are assigned to one of the clusters described by these parameters.

We note that the GMM model in Eq. (15.11) is actually the same as Eq. (13.11) in the naive Bayes classification. These two methods are similar in the sense that each cluster or class C_{k} is modeled by a Gaussian N (m_{k}, Σ_{k}), weighted by P_{k}, and the model parameters m_{k} and Σ_{k}, as well as P_{k}, are to be estimated based on the given dataset.

However, the two methods are different in that the dataset X in the supervised naive Bayes method is labeled by y, which is not available in unsupervised GMM. Instead, here we introduce a latent or hidden variable z = [z_{1}, ⋯, z_{K}]^{T} as the cluster labeling of each ample x in the dataset X. All components of z_{n} are binary random variables z_{k} ∈ {0, 1}, of which only one is 1, e.g., z_{l} = 1, indicating x belongs to the lth cluster C_{l}, while all others are 0. As x must belong to one of the K clusters, these K binary variables add up to 1, i.e., ∑_{k=1}^{K} z_{k} = 1.

We consider the following probabilities defined for each of the K clusters C_{k} (k = 1, ⋯, K):

• The prior probability P_{k}, (k = 1, ⋯, K) for any unobserved data sample in the dataset to belong to C_{k}, they satisfy

∑_{k=1}^{K} P_{k} = 1  (15.13)

These K prior probabilities add up to 1, i.e, the K events z_{k} = 1 (k = 1, ⋯, K) are mutually exclusive and complementary, as any sample x belongs to one and only one of the K clusters.

• The probability distribution of all samples in cluster C_{k}, assumed to be a Gaussian:

p(x|z_{k} = 1, θ) = N (x, m_{k}, Σ_{k})  (15.14)

• The joint distribution of x and z_{k} = 1:

p(x, z_{k} = 1|θ) = p(x|z_{k} = 1, θ) P (z_{k} = 1) = P_{k}N (x, m_{k}, Σ_{k}) (15.15)

where P (z_{nk} = 1) = P_{k} is the probability for any x_{n} to belong to C_{k}.

Marginalizing this joint probability over the latent variable z_{k}, we get the Gaussian mixture model, the distribution p(x) of any sample x regardless to which cluster it belongs:

p(x|θ) = ∑_{k=1}^{K} p(x, z_{k} = 1fe|θ) = ∑_{k=1}^{K} p(x|z_{k} = 1, θ) P (z_{k} = 1) = ∑_{k=1}^{K} P_{k}N (x, m_{k}, Σ_{k})

(15.16)

Note that Eqs. (15.13) through (15.16) are the same as Eqs. (13.8) through (13.11) in the naive Bayes classifier, respectively.

All such probabilities defined for z_{k} = 1 can be generalized to z = [z_{1}, ⋯, z_{K}]^{T} for all K:

p(z|θ) = ∏_{k=1}^{K} P_{k}^{z_{k}}  (15.17)

p(x|z, θ) = ∏_{k=1}^{K} N (x, m_{k}, Σ_{k})^{z_{k}}  (15.18)

p(x, z|θ) = p(z) p(x|z, θ) = ∏_{k=1}^{K} (P_{k}N (m_{k}, Σ_{k}))^{z_{k}}  (15.19)

We further define Z = [z_{1}, ⋯, z_{N}] of which the nth column z_{n} = [z_{n1}, ⋯, z_{nK}]^{T} is the labeling of the nth sample x_{n} in the dataset X containing N i.i.d. samples. Note that here Z is defined in the same way as Y = [y_{1}, ⋯, y_{N}] in softmax regression (Section 7.2), both treated as the labeling of X, with the only difference that Y is provided in the training data available for a supervised method, but here Z is a latent variable, not available in the unlabeled dataset in unsupervised clustering.

For a specific sample x_{n} in the dataset, the joint probibility in Eq. (15.19) can be written as

p(x_{n}, z_{n}|θ) = ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}} ,  (n = 1, ⋯, N) (15.20)

based on which the likelihood function of the GMM model parameter θ can be expressed as

L(θ|X, Z) = p(X, Z|θ) = p ([x_{1}, ⋯, x_{N}], [z_{1}, ⋯, z_{N}]|m_{k}, Σ_{k}, P_{k} (k = 1, ⋯, K))
= ∏_{n=1}^{N} p(x_{n}, z_{n}|θ) = ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}}  (15.21)

and the log likelihood is:

l(θ|X, Z) = log L(θ|X, Z) = log p(X, Z|θ)
= log [∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}N (x_{n}, m_{k}, Σ_{k}))^{z_{nk}}]

= ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]  (15.22)

Similar to the general method of maximum likelihood estimation (MLE) that finds the optimal model parameter θ that maximizes the likelihood L(θ), here the GMM method finds the model parameter θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} that maximizes the expectation of the likelihood function l(θ|X, Z). However, as GMM is an unsupervised method, the ground truth labeling is not available in the given dataset, it has to be carried out in two separate steps. First, in the E-step the expectation of the log likelihood function is estimated based on the previously estimated model parameters; second, in the M-step the estimated parameters that maximize the expectation of the log likelihood are obtained and the previous estimation is updated. As these two steps are interdependent of each other, they need to be carried out alternatively in an iteration from some initial guess of the parameters. This process is called an EM method due to these E and M steps.

Now we consider the specific computations taking place in each of the two steps.

• E-step: Find the expectation of the log likelihood function.

We first find the posterior probability for an observed x_{n} to belong to cluster C_{k}, indicated by z_{nk} = 1 (and z_{nl} = 0 for all l ≠ k):

P_{nk} = P (z_{nk} = 1|x_{n}, θ) = \frac{p(x_{n}, z_{nk} = 1|θ)}{p(x_{n}|θ)} = \frac{P_{k} N (x_{n}; m_{k}Σ_{k})}{∑_{l=1}^{K} P_{l} N (x_{n}; m_{l}Σ_{l})}
(n = 1, ⋯, N; k = 1, ⋯, K)  (15.23)

As x_{n} has to belong to one of the K clusters, these K posterior probabilities add up to 1, same as the prior probabilities P (z_{k} = 1) = P_{k}:

∑_{k=1}^{K} P (z_{k} = 1) = ∑_{k=1}^{K} P_{k} = 1,  ∑_{k=1}^{K} P (z_{nk} = 1|x_{n}, θ) = ∑_{k=1}^{K} P_{nk} = 1

(15.24)

Once this posterior P_{nk} = p(z_{nk} = 1|x_{n}, θ) is calculated based on Eq. (15.23) for all n = 1, ⋯, N), they are used to assign x_{n} to cluster C_{k} if P_{nk} = max_{l} P_{nl}.

We note that the definition of P_{nk} is similar to the softmax function s_{nk} with the only difference that here the weighted Gaussian function are used instead of the softmax function used in softmax regression.

We also note that the posterior probability P_{nk} defined above represents a soft decision, in the sense that it is possible for any x_{n} to belong to any C_{k} with probability 0 < P_{nk} < 1 for all k = 1, ⋯, K, instead of a hard decision, in the sense that x_{n} belongs to only one specific C_{k} with P_{nk} = 1, while P_{nl} = 0 for all l ≠ k, as in K-means clustering.

We can now find the expectation of the log likelihood with respect to the latent variables in Z:

E_{Z} (log L(θ|X, Z)) = E_{Z} [∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]]

= ∑_{n=1}^{N} ∑_{k=1}^{K} E_{Z}(z_{nk}) [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})]

= ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] (15.25)

The last equality is due to the following:

E_{z}(z_{nk}) = 1 P (z_{nk} = 1|x_{n}) + 0 P (z_{nk} = 0|x_{n}) = P (z_{nk} = 1|x_{n}) = P_{nk}

(15.26)

• M-step: Find the optimal model parameters that maximize the expectation

of the log likelihood in Eq. (15.25).

We first set to zero the derivatives of the expectation of the log likelihood with respect to each of the parameters in θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)}, and then solve the resulting equations to get the optimal parameters.

– Find P_{k}:

As all K of P_{k}’s need to satisfy the constraint ∑_{k=1}^{K} P_{k} = 1, we first construct the Lagrangian function composed of an extra Lagrange multiplier term for the constraint as well as the log likelihood as the objective function:

L(θ, λ) = ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] + λ (∑_{k=1}^{K} P_{k} − 1)

(15.27)

and set it derivative with respect to P_{k} to zero:

\frac{∂}{∂P_{k}} L(θ, λ) = \frac{∂}{∂P_{k}} [∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + log N (x_{n}, m_{k}, Σ_{k})] + λ (∑_{k=1}^{K} P_{k} − 1)]

= ∑_{n=1}^{N} P_{nk} \frac{1}{P_{k}} + λ = 0  (15.28)

Multiplying both sides by P_{k}, we get

∑_{n=1}^{N} P_{nk} + P_{k}λ = N_{k} + P_{k}λ = 0  (15.29)

where we have defined

N_{k} = ∑_{n=1}^{N} P_{nk}  (15.30)

that satisfies

∑_{k=1}^{K} N_{k} = ∑_{k=1}^{K} (∑_{n=1}^{N} P_{nk}) = ∑_{n=1}^{N} (∑_{k=1}^{K} P_{nk}) = ∑_{n=1}^{N} 1 = N (15.31)

Summing Eq. (15.29) over all k = 1, ⋯, K, we get

∑_{k=1}^{K} (N_{k} + P_{k}λ) = ∑_{k=1}^{K} N_{k} + λ (∑_{k=1}^{K} P_{k}) = N + λ = 0 (15.32)

Substituting λ = −N back into Eq. (15.29), we get the expression for the prior

p(z_{k} = 1) = P_{k} = \frac{N_{k}}{N} = \frac{1}{N} ∑_{n=1}^{N} P_{nk}  (15.33)

This is the same as the prior P_{k} in Eq. (13.8) in naive Bayes classification, but here N_{k} defined in Eq. (15.30) is the sum of the probabilities for all N data samples to belong to C_{k}, instead of the number of data samples in C_{k} (unknown in this unsupervised case).

– Find m_{k}:

\frac{∂}{∂m_{k}} E_{Z}(log L(θ|X, Z)) = \frac{∂}{∂m_{k}} [∑_{n=1}^{N} ∑_{l=1}^{K} P_{nl} [log P_{l} + log N (x_{n}, m_{l}, Σ_{l})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} log N (x_{n}, m_{k}, Σ_{k})

=^{∗} ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} [−\frac{1}{2}(x_{n} − m_{k})^{T} Σ_{k}^{−1}(x_{n} − m_{k})]

= \frac{1}{2} ∑_{n=1}^{N} P_{nk}Σ_{k}^{−1}(x_{n} − m_{k}) = 0  (15.34)

Note that here =^{∗} indicates that we have neglected the scaling coefficient (2π)^{−d/2} |Σ_{k}|^{−1/2} of the Gaussian, which is independent of m_{k}. Multiplying 2Σ_{k} on both sides, we get

∑_{n=1}^{N} P_{nk}(x_{n}−m_{k}) = ∑_{n=1}^{N} P_{nk}x_{n}−∑_{n=1}^{N} P_{nk}m_{k} = ∑_{n=1}^{N} P_{nk}x_{n}−N_{k}m_{k} = 0

(15.35)

Solving for m_{k} we get

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}  (15.36)

– Find Σ_{k}:

\frac{∂}{∂Σ_{k}} E_{Z}(log L(θ|X, Z)) = \frac{∂}{∂Σ_{k}} [∑_{n=1}^{N} ∑_{l=1}^{K} P_{nl} [log P_{l} + log N (x_{n}, m_{l}, Σ_{l})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂Σ_{k}} log N (x_{n}, m_{k}, Σ_{k})

= −\frac{1}{2} ∑_{n=1}^{N} P_{nk} [\frac{∂}{∂Σ_{k}} log |Σ_{k}| + \frac{∂}{∂Σ_{k}} (x_{n} − m_{k})^{T} Σ_{k}^{−1}(x_{n} − m_{k})]

= −\frac{1}{2} ∑_{n=1}^{N} P_{nk} [Σ_{k}^{−1} − Σ_{k}^{−1}(x_{n} − m_{k})(x_{n} − m_{k})^{T} Σ_{k}^{−1}] = 0  (15.37)

Here we have used the following facts (for more details see Matrix

         Cookbook https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf)

\frac{d}{dA} log |A| = (A^{−1})^{T} , \frac{d}{dA} (a^{T} A^{−1}b) = −(A^{−1})^{T} ab^{T} (A^{−1})^{T}

(15.38)

Pre and post multiplying Σ_{k} on both sides of the equation above we get

∑_{n=1}^{N} P_{nk} (Σ_{k} − (x_{n} − m_{k})(x_{n} − m_{k})^{T} ) = 0  (15.39)

Solving for Σ_{k}, we get

Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}(x_{n} − m_{k})(x_{n} − m_{k})^{T}  (15.40)

We note that all model parameters in θ = {P_{k}, m_{k}, Σ_{k}, (k = 1, ⋯, K)} found respectively in Eqs. (15.33), (15.36), and (15.40) in the M-step depend on P_{nk} in Eq. (15.23) in the E-step, which in turn is a function of these parameters, i.e., the E-step and M-step need to be carried out in an alternative and iterative fashion from some initial values of the parameters until convergence.

In summary, here are the steps of GMM clustering algorithm:

1. Initialize means m_{k}, covariance Σ_{k} and coefficient P_{k}.

2. The E-step:

Find P_{nk} for all N data points and all K clusters and then N_{k}:

P_{nk} = P (r_{k} = 1|x_{n}) = \frac{P_{k} N (x_{n}; m_{k}, Σ_{k})}{∑_{l=1}^{K} P_{l} N (x_{n}; m_{l}, Σ_{l})},  N_{k} = ∑_{n=1}^{N} P_{nk} (15.41)

3. The M-step:

Recalculate the parameters that maximize the likelihood function:

P_{k} = \frac{N_{k}}{N}
m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}

Σ_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}(x_{n} − m_{k})(x_{n} − m_{k})^{T}  (15.42)

4. Terminate the iteration if the parameters have converged, i.e., the difference between their values of two consecutive iterations is sufficiently small. Otherwise go back to step 2. The probability for each sample x_{n} to belong to cluster C_{l} is p_{nl} = P (z_{nl} = 1|x_{n}, θ), and it is therefore assigned to C_{k} if P_{nk} = max_{l} p_{nl}.

The Matlab code below is the essential part of the algorithm including the initialization of parameters and the iteration loop. The function mvnpdf(X,mk(:,k),Ck(:,:,k)) evaluate the Gaussian pdf of the kth cluster N (x_{n}; m_{k}, Σ_{k}) at all N samples in dataset X.

[N d]=size(X);               % N samples of d dimensions
mk=zeros(d,K);               % old mean vectors
Mk=zeros(d,K);               % new mean vectors
Ck=zeros(d,d,K);             % covariance matrices
p=zeros(N,K);                % posterior
Pnk=zeros(N,K);              % Pnk
Pk=zeros(K,1);               % prior Pk=Nk/N
Nk=zeros(1,K);               % number of samples in kth cluster
for k=1:K                    % initial parameters for kth cluster
    sd=randi(N,1,2*d);       % random indices of subsamples
    mk(:,k)=mean(X(sd,:))';  % mean of these samples
    Ck(:,:,k)=cov(X(sd,:));  % covariance of these samples
    Pk(k)=1/K;               % prior probability
end
er=1;
while er>10^(-4)
    for k=1:K                % for each of K clusters
        p(:,k)=Pk(k)*mvnpdf(X,mk(:,k),Ck(:,:,k));  % weighted posterior
    end
    D=sum(p,2);              % sum over all K terms
    er=0;
    for k=1:K                % update parameters for kth cluster
        Pnk(:,k)=p(:,k)./D;  %
        Nk(k)=sum(Pnk(:,k)); % Nk
        Pk(k)=Nk(k)/N;       % Pk

        if Nk(k)>0
            Mk(:,k)=sum(X.*repmat(Pnk(:,k),[1,d]),1)/Nk(k);   % mean
            Xm=X-repmat(Mk(:,k)',[N,1]);
            Ck(:,:,k)=Xm'*(Xm.*repmat(Pnk(:,k),[1,d]))/Nk(k); % covariance
        end
        er=er+norm(mk(:,k)-Mk(:,k));  % difference between two iterations
    end
    mk=Mk;                      % keep iterating
end

We can show that the K-means algorithm is actually a special case of the EM algorithm, when all covariance matrices are the same Σ_{k} = σ^{2}I, and σ^{2} approaches to zero. In this case we have:

p(x|z_{k} = 1, θ) = N (x|m_{k}, σ^{2}I) = \frac{1}{(2π)^{d/2}σ} exp (−\frac{1}{2σ^{2}} ||x − m_{k}||^{2}) (15.43)

and the probability for any x_{n} ∈ X to belong to cluster C_{k} is:

P_{nk} = P (z_{k} = 1|x_{n}, θ) = \frac{P_{k}N (x_{n}; m_{l}Σ_{l})}{∑_{l=1}^{K} P_{l}N (x_{n}; m_{l}Σ_{l})} = \frac{P_{k} exp(−||x_{n} − m_{k}||^{2}/2σ^{2})}{∑_{l=1}^{K} P_{l} exp(−||x_{n} − m_{l}||^{2}/2σ^{2})}

(15.44)

When σ^{2} → 0, all terms in the denominator approach to zero, but the one with minimum ||x − m_{l}|| approaches to zero most slowly, and becomes the dominant term of the denominator. If the numerator happens to be this term, i.e., k = l, then P_{nk} = 1; otherwise the numerator approaches zero and P_{nk} = 0. Now P_{nk} defined above becomes:

lim_{σ^{2}→0} P_{nk} = lim_{σ^{2}→0} \frac{P_{k} exp(−||x_{n} − m_{k}||^{2}/2σ^{2})}{∑_{l=1}^{K} P_{l} exp(−||x_{n} − m_{l}||^{2}/2σ^{2})}
= \{ \begin{matrix}1 & if ||x_{n} − m_{k}|| = min_{l} ||x_{n} − m_{l}|| \\ 0 & otherwise \end{matrix}  (15.45)

Now the posterior probability 0 < P_{nk} < 1 defined in Eq. (15.23) for a soft decision becomes a binary value P_{nk} ∈ {0, 1} for a hard binary decision to assign x_{n} to C_{k} with the smallest distance. Also N_{k} = ∑_{n=1}^{N} P_{nk} defined in Eq. (15.30) as the sum of the posterior probabilities for all N data points to belong to C_{k} becomes N_{k} as the number of data samples assigned only to C_{k}. In other words, now the probabilistic EM method based on both m and Σ becomes the deterministic K-means method based on m only.

We can also make a comparison between the GMM method for unsupervised clustering and the softmax regression for supervised classification. First, the latent variables Z = [z_{1}, ⋯, z_{N}] in GMM play a similar role as the labeling Y = [y_{1}, ⋯, y_{N}] in softmax regression multiclass classification. However, the difference is that Y is explicitly given in the training set for a supervised classification, while Z is hidden for an unsupervised clustering analysis. Second, we note that the probability P_{nk} = p(z_{nk} = 1|x_{n}, θ) given in Eq. (15.23) is similar to the softmax function ϕ_{nk} = P (y′ = k|x_{n}) in the softmax method in terms of their form, with the only difference that the Gaussian function is used for GMM while the exponential function is used for softmax.

Example 15.3 Both the K-means and GMM clustering methods are tested based on the same synthetic dataset generated by the following Matlab code

    mk=zeros(d,3);
    Ck=zeros(d,d,3);
    Ck(:,:,1)=[1 1.5; 1.5 5];   mk(:,1)=[5; 4];
    Ck(:,:,2)=[3 -1.5; -1.5 1]; mk(:,2)=[6; 5];
    Ck(:,:,3)=[2 1; 1 2];       mk(:,3)=[10; 2];
    Ki=[200 200 200];
    X1=mvnrnd(mk(:,1)',Ck(:,:,1),Ki(1));
    X2=mvnrnd(mk(:,2)',Ck(:,:,2),Ki(2));
    X3=mvnrnd(mk(:,3)',Ck(:,:,3),Ki(3));

This code generates three clusters of normally distributed data points. Note that the means of the first two clusters are very close to each other, but their covariances are such that the their distributions overlap with each other but are in very different orientations.

The intermediate results after each iteration are shown in the panels in Fig. 15.5 for the K-means method and Fig. 15.6 for the EM clustering method. The data points are color coded according to their class identity (ground truth not used in clustering), and the contour lines for each of the clusters are also shown based on the mean and covariance of all sampled assigned to each cluster in the current iteration. Comparing the two sets of results obtained by the two methods, we see that the K-means method cannot separate the first two clusters (in red and green), while the third cluster (in blue) is separated incorrectly into two clusters. On the other hand, the GMM method can correctly separate all three clusters.

Example 15.4 The two clustering methods are also applied to the 4-D Iris dataset containing three classes each of 50 sample vectors. The PCA method is used to visualize the first two principal components, as shown in Figs. 15.7 and 15.8. Also, as can be seen from their confusion matrices in Eq. (15.46), the error rate of the EM method is 5/150, much smaller than 18/15 of the K-means method.

\begin{matrix}K − means \\ 0 & 0 & 50 \\ 50 & 0 & 0 \\ 18 & 32 & 0 \end{matrix}  \begin{aligned}& EM \\ 0 & 0 & 50 \\ 45 & 5 & 0 \\ 0 & 50 & 0 \end{aligned}

Figure 15.5 Clustering by K-Means Method (first 8 iterations)

Figure 15.6 Clustering by Gaussian Mixture Model (first 12 iterations)

15.3 Bernoulli Mixture Model

If the data points are binary, i.e., each data point is treated as a binary random variable that takes either of two values such as 1 and 0 with probabilities μ and 1−μ, then the assumed Gaussian distribution is no longer valid and the Gaussian

Figure 15.7 Clustering of Iris Dataset by K-Means Method

Figure 15.8 Clustering of Iris Dataset by Gaussian Mixture Model

mixture model is not suitable. In this case, we can modify the EM method for clustering based on Bernoulli mixture model (BMM) instead of GMM.

The probability mass function (pmf) of the Bernoulli distribution can be used instead:

B(x, μ) = μ^{x}(1 − μ)^{1−x} = \{ \begin{matrix}μ & if x = 1 \\ 1 − μ & if x = 0 \end{matrix}  (15.46)

The mean and variance of x are

E[x] = 1 P (x = 1) + 0 P (x = 0) = 1 μ + 0 (1 − μ) = μ  (15.47)
V ar[x] = E[(x − E[x])^{2}] = E[x^{2}] − E[x]^{2}
= 1^{2} P (x = 1) + 0^{2} P (x = 0) − μ^{2} = μ − μ^{2} = μ(1 − μ) (15.48)

Note that the variance is solely determined by the mean E[x] = μ it does not need to be estimated separately, therefore the method based on BMM is simpler than that based on GMM.

A set of d independent binary variables can be represented as a random vector x = [x_{1}, ⋯, x_{d}]^{T} with mean vector and covariance matrix as shown below:

E[x] = m = [μ_{1}, ⋯, μ_{N}]^{T}  (15.49)
Cov[x] = Σ = diag(μ_{i}(1 − μ_{i})) = [ \begin{matrix}μ_{1}(1 − μ_{1}) & & & & 0 \\ & . & . & . \\ 0 & & & & μ_{d}(1 − μ_{d}) \end{matrix} ](15.50)

The off-diagonal covariances are zero as the d variables are assumed to be independent of each other. The joint pmf of the binary random vector x is

B(x, m) = ∏_{i=1}^{d} B(x_{i}|μ_{i}) = ∏_{i=1}^{d} μ_{i}^{x_{i}}(1 − μ_{i})^{1−x_{i}}  (15.51)

and the log pmf:

log B(x, m) = log (∏_{i=1}^{d} B(x_{i}, μ_{i})) = ∑_{i=1}^{d} [x_{i} log μ_{i} + (1 − x_{i}) log(1 − μ_{i})]

(15.52)

Similar to the Gaussian mixture model, the Bernoulli mixture model of K multivariate Bernoulli distributions is defined as:

p(x|θ) = p(x|m_{k}, P_{k}, (k = 1, ⋯, K)) = ∑_{k=1}^{K} P_{k}B(x, m_{k})

= ∑_{k=1}^{K} P_{k} ∏_{i=1}^{d} μ_{ki}^{x_{i}}(1 − μ_{ki})^{1−x_{i}}  (15.53)

where θ = {m_{k}, P_{k}, (k = 1, ⋯, K)} denotes all parameters of the mixture model to be estimated based on the given dataset. The mean of this mixture model is

m = E[x] = ∑_{k=1}^{K} P_{k}E_{k}[x] = ∑_{k=1}^{K} P_{k}m_{k}  (15.54)

Also similar to the Gaussian mixture model, we introduce a set of K latent binary random variables z = [z_{1}, ⋯, z_{K}]^{T} with binary components z_{k} ∈ {0, 1} satisfying ∑_{k=1}^{K} z_{k} = 1, and get the prior probability of z, the conditional probability of x given z, and the joint probability of x and z as the following

p(z|θ) = ∏_{k=1}^{K} P_{k}^{z_{k}}  (15.55)

p(x|z, θ) = ∏_{k=1}^{K} B(x, m_{k})^{z_{k}}  (15.56)

p(x, z|θ) = p(z|θ) p(x|z, θ) = ∏_{k=1}^{K} (P_{k} B(x, m_{k}))^{z_{k}}  (15.57)

Given the dataset X = [x_{1}, ⋯, x_{N}] containing N i.i.d. samples, we introduce the corresponding latent variables in Z = [z_{1}, ⋯, z_{N}], of which each z_{n} = [z_{n1}, ⋯, z_{nK}]^{T} is for the labeling of x_{n}. Then we can find the likelihood function of the Bernoulli mixture model parameters θ = {P_{k}, m_{k}, (k = 1, ⋯, K)}:

L(θ|X, Z) = p(X, Z|θ) = p([x_{1}, ⋯, x_{N}], [z_{1}, ⋯, z_{N}]|m_{k}, P_{k}(k = 1, ⋯, K))
= ∏_{n=1}^{N} p(x_{n}, z_{n}|θ) = ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}B(x_{n}, m_{k}))^{z_{nk}} (15.58)

and the log likelihood function:

log L(θ|X, Z) = log p(X, Z|θ) = log ∏_{n=1}^{N} ∏_{k=1}^{K} (P_{k}B(x_{n}, m_{k}))^{z_{nk}}

= ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log B(x_{n}, m_{k})]  (15.59)

Based on the same EM method used in Gaussian mixture model, we can find the optimal parameters that maximize the expectation of the log likelihood function in the following two steps:

• E-step: Find the expectation of the likelihood function.

We first find the posterior probability for any sample x_{n} to belong to cluster C_{k}, denoted by P_{nk}:

P_{nk} = P (z_{nk} = 1|x_{n}, θ) = \frac{p(x_{n}, z_{nk} = 1|θ)}{p(x_{n}|θ)} = \frac{P_{k} B(x_{n}; m_{k})}{∑_{l=1}^{K} P_{l} B(x_{n}; m_{l})}
(n = 1, ⋯, N; k = 1, ⋯, K)  (15.60)

which is the expectation of z_{nk} (same as Eq. (15.26)):

E[z_{nk}] = 1 P (z_{nk} = 1|x_{n}) + 0 P (z_{nk} = 0|x_{n}) = P (z_{nk} = 1|x_{n}) = P_{nk}

(15.61)

Now we can find the expectation of the log likelihood with respect to the latent variables in Z:

E_{z} (log L(θ|X, Z)) = E_{z} ∑_{n=1}^{N} ∑_{k=1}^{K} z_{nk} [log P_{k} + log B(x_{n}, m_{k})]

= ∑_{n=1}^{N} ∑_{k=1}^{K} E[z_{nk}] [log P_{k} + log ∏_{i=1}^{d} μ_{ki}^{x_{ni}}(1 − μ_{ki})^{1−x_{ni}}]

= ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]] (15.62)

• M-step: Find the optimal model parameters that maximize the expectation

of the log likelihood function.

We first set to zero the derivatives of the expectation of the log likelihood with respect to each of the parameters in θ = {P_{k}, m_{k} (k = 1, ⋯, K)}, and then solve the resulting equations to get the optimal parameters.

– Find P_{k}: same as in the case of the GMM model:

P_{k} = \frac{N_{k}}{N} = \frac{1}{N} ∑_{n=1}^{N} P_{nk}  (15.63)

– Find m_{k}:

\frac{∂}{∂m_{k}} E_{Z} (log p(X, Z|θ))

= \frac{∂}{∂m_{k}} ∑_{n=1}^{N} ∑_{k=1}^{K} P_{nk} [log P_{k} + ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]]

= ∑_{n=1}^{N} P_{nk} \frac{∂}{∂m_{k}} ∑_{i=1}^{d} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})] = 0 (15.64)

The ith component of the equation is

∑_{n=1}^{N} P_{nk} \frac{d}{dμ_{ki}} [x_{ni} log μ_{ki} + (1 − x_{ni}) log(1 − μ_{ki})]

= ∑_{n=1}^{N} P_{nk} (\frac{x_{ni}}{μ_{ki}} − \frac{1 − x_{ni}}{1 − μ_{ki}} ) = 0  (15.65)

i.e.,

(1 − μ_{ki}) ∑_{n=1}^{N} P_{nk}x_{ni} = μ_{ki} ∑_{n=1}^{N} P_{nk}(1 − x_{ni}) = μ_{ki}N_{k} − μ_{ki} ∑_{n=1}^{N} P_{nk}x_{ni}

(15.66)

Solving for μ_{ki} we get

μ_{ki} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{ni} (i = 1, ⋯, d)  (15.67)

or, in vector form,

m_{k} = \frac{1}{N_{k}} ∑_{n=1}^{N} P_{nk}x_{n}  (15.68)

The Matlab code shown below is the iteration part of the algorithm. Same as in the code the Gaussian mixture model, all parameters are initialized in a similar manner.

er=1;
while er<0.01
for k=1:K
    p(:,k)=pi(k)*Bpdf(X,mk(:,k)');
end
er=0;
D=sum(p,2);
for k=1:K
    Pnk(:,k)=p(:,k)./D;
    Nk(k)=sum(Pnk(:,k));
    pi(k)=Nk(k)/N;

Figure 15.9 Clustering of Handwritten Digits (left K = 10, right K = 12)

      if Nk(k)>0
          Mk(:,k)=sum(X.*repmat(Pnk(:,k),[1,d]),1)/Nk(k);
          Xm=X-repmat(Mk(:,k)',[N,1]);
      end
      er=er+norm(mk(:,k)-Mk(:,k));
  end
end

Example 15.5 The clustering algorithm based on the mixture of Bernoulli model is applied to the dataset of handwritten digits after the gray scale images (in the range from 0 to 255) are converted to binary data by a threshold value (e.g., 128). Once the iteration is converged, the final clustering result is shown in the form of the means vectors of the K clusters (no longer binary) in Fig. 15.9 with K = 10 (left) and K = 12 (right).

Problems

1. Implement K-means clustering algorithm and apply it to the iris data. Try different values for K, such as K = 2 and K = 4 in addition to the ground truth value K = 3.

Show your results as a K by K confusion matrix and the error rate. Also visualize your result in both 2-D and 3-D spaces after the 4-D dataset is KLT transformed into a lower dimensional space. Color code all data points according to the cluster they are assigned to.

2. Repeat the above for the handwritten digit dataset. Again try different values

of K, such as 9 and 11, in addition to the ground truth value K = 10.

Further more, convert the K mean vectors of all samples assigned to the same clusters into 16 × 16 images, and display these images to see what the clusters actually represents, and if they each resemble one of the digits, same as those shown in Example 15.2.

3. Implement GMM clustering and apply it with the K-means method to the dataset used in Example 15.3 (generated by the code provided in the example). Visualize the results by both GMM and K-means methods will all data points color coded, and compare the confusion matrices of both methods to confirm GMM achieves much better results than K-mean in this particular case.

4. Apply the GMM method to both the iris and handwritten datasets. For each dataset, show your clustering result in confusion matrix and visualize the color coded data points in 2-D and 3-D spaces. Compare your results with those obtained by the K-means method in the first two problems.

5. Implement clustering based on mixture of Bernoulli model, then apply it to the binary version of the handwritten digit dataset obtained by thresholding all samples and converting them into binary numbers 0 or 1. Compare your results with K = 10 with those shown in Fig. 15.9

16 Hierarchical Classifiers

Both supervised classification and unsupervised clustering can be carried out hierarchically in the form of a dendrogram, a tree-like structure, very much like the classifications of different taxonomic ranks in biology (domain, kingdom, phylum, class, order, family, genus, and species). Such a hierarchical classification or clustering method is essentially a divide-and-conquer process by which a complex problem is converted into a sequence of simpler subproblems that can be solved more effectively and efficiently. Certain advantages can be gained by carrying out classification or clustering in a hierarchical manner, in comparison to most of the algorithms previously discussed all attempting to achieve the goal in a single step.

Before the considering various hierarchical algorithms specifically, we note that as a general method for classification and clustering, a decision tree may grow very deep to accommodate certain detailed variations in the data, and consequently overfitting the data. This problem can be addressed by the method of random forest which takes the average of multiple decision trees each trained on different subsets of the training dataset. Random forest is similar to the K-fold cross-validation in the sense that the given dataset is subdivided into multiple parts for more reliable outcomes.

16.1 Bottom-Up vs Top-Down Methods

The tree structure for a hierarchical method is typically constructed in either bottom-up fashion by merging subgroups of data points into bigger groups, or top-down fashion by splitting big groups into smaller subgroups. In the following, we will concentrate on how the merging or splitting is specifically carried out when applied to a classification problem with K classes, to be considered in Section 16.2. The resulting binary tree, typically upside down, has 2K −1 nodes, including K leaf nodes each for one of the classes, and K − 1 non-leaf nodes, at each of which two subgroups are either merged into one group in the bottom-up case, or group is split into two subgroups in the top-down case.

Either of the top-down splitting and bottom-up merging methods can also be used to construct a hierarchical structure for unsupervised clustering if applied to all N individual unlabeled data samples in the given dataset, instead of K labeled classes, as we will see in Section 16.3.

• Bottom-Up method:

Initially, every class composed of data points with the same class identity is treated as subgroup represented by one of the K leaf nodes at the bottom of the binary tree. These subgroups are then merged two at a time to form progressively larger groups. This process is carried out recursively K − 1 times until eventually all K classes are merged into a single super group at the root node of the tree. In the following discussion, we assume the dataset contains in total N data samples of which n_{k} are labeled to belong to class C_{k}, (k = 1, ⋯, K). The steps for this bottom-up merging method is shown below. 1.Compute the K(K − 1)/2 pairwise Bhattacharyya distances between every two classes C_{i} and C_{j}:

d_{B}(C_{i}, C_{j}) = \frac{1}{4}(m_{i}−m_{j})^{T} [\frac{Σ_{i} + Σ_{j}}{2} ]^{−1} (m_{i}−m_{j})+log [\frac{|\frac{Σ_{i}+Σ_{j}}{2} |}{(|Σ_{i}| |Σ_{j}|)^{1/2}} ]

(16.1)

2.Merge the two classes corresponding to the smallest d_{B} to form a new class C_{k} = C_{i} ∪ C_{j}, and compute its mean

m_{k} = \frac{1}{n_{i} + n_{j}} [n_{i}m_{i} + n_{j}m_{j}]  (16.2)

and covariance

Σ_{k} = \frac{1}{n_{i} + n_{j}} [n_{i}(Σ_{i}+(m_{i}−m_{k})(m_{i}−m_{k})^{T})+n_{j}(Σ_{j}+(m_{j}−m_{k})(m_{j}−m_{k})^{T})]

(16.3)

Delete the old classes C_{i} and C_{j}. Now there are K − 1 classes left. 3.Repeat the previous steps to find all pairwise distances for the remaining classes, and merge the two with minimum distance, until eventually all classes are merged into a single group containing all K classes, the binary tree structure is thus obtained.

• Top-Down method:

Initially, all K classes are treated as members of a single super group at the root node on top of the tree, which is then split into two subgroups each represented as a node at a lower level of the tree. This subdivision is carried out recursively until eventually each subgroup contains a single class represented by one of the leaf nodes at the bottom of the tree. This tree structure can be constructed in either breadth-first manner based on a queue, or depth-first manner based on a stack, which can be most conveniently implemented recursively.

Various methods can be used to subdivide a group into two subgroups. For example, the K-means method can be used with K = 2. Alternatively, the binary subdivision can be carried out in the 1-D space along the direction of the principal component based on the KLT method, applied to either the covariance matrix of the entire dataset, or the between-class scatter matrix (Eq. (9.20)). All samoples are projected onto this direction and then subdivided into two parts with maximum Bhattacharyya distance, as shown in the following steps. 1.Compute the covariance matrix Σ of all N data points or thebetweenclass scatter matrix S_{B} of all K classes, find its maximum eigenvalue λ_{i} and the corresponding eigenvectors v_{i}; 2.Project all data points onto v_{1}:

y_{n} = x_{n}^{T} v (n = 1, ⋯, N)  (16.4)

3.Sort all data points {y_{1}, ⋯, y_{N}} along this 1-D space and partition them into two subgroups with maximum Bhattacharyya distance. 4.Carry out the steps above recursively to each of the two subgroups, until eventually every subgroup contains only one class.

The hierarchical structure so constructed by either method can then be used as either a decision tree classifier for supervised classification or unsupervised clustering as respectively discussed in the following two sections.

16.2 Binary Hierarchical Classification

The hierarchical classification method is especially suitable when the dimension of the feature space is high and the number of classes is large. For a single-level classifier, it may be difficult to select a subset of the large number of available features good for separating all classes in a single step. On the other hand, for a hierarchical classifier it is possible to select a small subset of features that are most relevant and suitable to distinguish two subgroups at each node. Although this is a multi-step process, it can be carried out effectively and efficiently due to the low computational complexity of the binary classification.

Specifically, any unlabeled sample to be classified enters the tree structure at the root node, where it is classified into either the left or right subgroup, and this binary classification is further carried out recursively at each of the tree nodes along a certain branch of the tree, until eventually reaching one of the K leaf nodes representing class identity of the data sample. Here are the specific steps:

1. According to the specific classification method used, find the discriminant functions D_{l}(x) and D_{r}(x) for the two subgroups based on the training data.

2. Select the best d < D features most suitable for separating the two groups G_{l} and G_{r}, based on any of the feature selection methods such as those listed below:

• Choosing d features directly from the D original ones using between-class distance (e.g., Bhattacharrya distance) as the criterion, • Carry out KLT based on the between-class scatter matrix S_{B} and use the

first d principal components for the binary classification. As here only two groups of classes need to be distinguished, the number of selected features can be expected to be small.

3. Any unlabeled pattern x enters the classifier at the root of the tree and is classified to either the left or the right sub-group of the node according to the discriminant function

x ∈ \{ \begin{matrix}G_{l} & if D_{l}(x) > D_{r}(x) \\ G_{r} & if D_{r}(x) < D_{l}(x) \end{matrix}  (16.5)

This process is carried out recursively at each of the subsequent nodes until eventually x reaches one of the leaf nodes corresponding to a single class, to which the sample x is therefore classified.

The Matlab function listed below shows how the top-down splitting method can be implemented recursively as a depth-first search process. Here X is a 2-D data array containing N column vectors each for one of the N data samples labeled as one of the K classes, M is a 2-D array containg K columns each for the mean vector of one of the K classes, and Cov is a 3-D array containing the covariance matrices of the K classes (both pre-calculated in the main function). Also, p is a 1-D vector for the parent node of each of the 2K − 1 tree nodes, to be used for visualizing the tree using the Matlab function treeplot.

function TDclassify()         % top-down tree construction
global X M Cov K p nc
[d N]=size(X);            % dimension and size of data
gid=1:K;                  % list of all K classes
nc=K;                     % initialize current node
p=zeros(1,2*K-1);         % parent list for all tree nodes;
f=0;                      % parent node is always 0
DFdivideC(gid,f);         % top-down binary tree construction
treeplot(p)               % plot the binary tree
[x,y]=treelayout(p);
text(x+0.02,y, compose('%d',[1:2*K-1])) % label all tree nodes
end
function DFdivideC(gid,f)     % depth-first search recursion
global X K M Cov p nc     % global variables shared by all functions
if length(gid)==1         % leaf-node, terminate recursion
     return
end
nc=nc+1;                  % current number of nodes
p(nc)=f;                  % assign parent
f=nc;                     % parent node for its children
[lid rid]=twoMeansC(gid); % split gid into two parts lid and rid

if length(lid)==1         % left branch is a leaf node
p(lid)=nc;
if length(rid)==1     % leaf node on right
    p(rid)=nc;
end
else                      % left branch is non-leaf node
DFdivideC(lid,f)      % recursion of the left branch
end
DFdivideC(rid,f)          % recursion of the right branch
end

The function below subdivides the given list of groups in gid into two subgroups lid on the left and rid on the right, based on the method of K-means clustering with K = 2.

function [lid rid]=twoMeansC(gid) % 2-means clustering
global X M Cov Nk
K=length(gid);          % number of classes in input
if K==2
lid=gid(1);
rid=gid(2);
return
end
m1=M(:,gid(1));         % first 2 classes as seeds for 2 means
m2=M(:,gid(2));
er=inf;
while er>0              % clustering iteration
gid1=[]; cid1=[];   % group IDs and classes IDs for group 1
gid2=[]; cid2=[];   % group IDs and classes IDs for group 2
m1o=m1;  m2o=m2;    % old means
for l=1:K           % for each of K classes
    k=gid(l);       % kth cluster ID
    m=M(:,k);       % mean of cluster k
    c=Cov(:,:,k);   % covariance of cluster k
    d1=distM(m1o,m,c);
    d2=distM(m2o,m,c);
    if d1 < d2
        gid1=[gid1 k];              % assign class k to group 1
        cid1=[cid1 cid(k,1:Nk(k))]; % assign samples to group 1
    else
        gid2=[gid2 k];
        cid2=[cid2 cid(k,1:Nk(k))];
    end
end
m1=mean(X(:,cid1)')'; % two new means

    m2=mean(X(:,cid2)')';
    er=norm(m1-m1o)+norm(m2-m2o);
end
lid=gid1; rid=gid2;
if length(gid2)==1        % single node always on left branch
    lid=gid2; rid=gid1;
end
end

The bottom-up method for the binary tree construction keeps merging two groups of classes with minimum Bhattacharyya distance recursively from the leaf nodes each representing a single class all the way up to the root node representing a super group containing all classes. This process can be implemented by the Matlab function listed below, where Nk is a list containing the number of samples in each of the K classes. The main loop is carried out K − 1 times to keep merging two subgroups (to be deleted) with minimum distance into a group (to be created). After K − 1 such merge operations, all K classes are merged into a single super group at the root node. Again, the parent nodes of the 2K − 1 tree nodes are stored in a list p for the visualization of the binary tree.

function BUtree()           % bottom-up tree construction
global X K Nk M Cov p groups
[d N]=size(X);          % number of classes
cid=zeros(2*K,N);       % sample IDs in each cluster
del=ones(1,2*K);        % deleted clusters
p=zeros(1,2*K-1);       % parents list;
Ck=zeros(1,2*K);        % number of samples per group
Ck(1:K)=Nk;             % for K leaf nodes
ik=0;
for k=1:K
    cid(k,1:Ck(k))=ik+1:ik+Ck(k);
    ik=ik+Ck(k);
end
M=zeros(d,2*K);         % means for K leaf nodes and K-1 nonleaf nodes
Cov=zeros(d,d,2*K);     % covariance of all 2K-1 nodes
k=K;
while k<2*K-1           % merge K-1 times
    for l=1:k           % get mean and cov for each group
        M(:,l)=mean(X(:,cid(l,1:Ck(l)))')';
        Cov(:,:,l)=cov(X(:,cid(l,1:Ck(l)))');
    end
    dmin=inf;
    for k1=2:k
        if del(k1)          % if first group is not deleted
            for k2=1:k1-1

                    if del(k2)  % if second group is not deleted
                        d=distB(M(:,k1),Cov(:,:,k1),M(:,k2),Cov(:,:,k2));
                        if dmin>d
                            dmin=d; i=k1; j=k2; % save min-distance so far
                        end
                    end
                end
            end
        end
        k=k+1;
        fprintf('(%2d,%2d) --> %2d\n',i,j,k)
        p(i)=k; p(j)=k;         % assign parent Ck to Ci and Cj
        cid(k,1:Ck(i)+Ck(j))=[cid(i,1:Ck(i)) cid(j,1:Ck(j))];
        Ck(k)=Ck(i)+Ck(j);
        del(i)=0; del(j)=0;     % delete Ci and Cj
    end
end

Example 16.1 Fig. 16.1 shows the binary tree structures created by both bottom-up (left) and top-down (left) methods applied to the dataset containing 10 classes of handwritten digits 0 through 9, as shown at the bottom of the 10 leaf nodes in both cases.

In the top-down method, node (K + 1) = 11 is the root node from which the super group containing all K classes is split recursively into the 10 leaf nodes each for one of the 10 classes, while in the bottom-up method, the 10 leaf nodes are recursively merged into a super group containing all 10 classes at node 2K − 1 = 19 as the root node. Comparing the trees constructed by the two methods, we see that they are similar in that all classes similar in shape are grouped together, e.g., digits 3, 5, and 8 form a subgroup at node 18 in the top-down tree and node 14 in the bottom-up tree, and digits 4, 7, and 9, form another subgroup at nodes 15 and node 13 in the respective trees, while digits 0, 2, and 6 with dissimilar shapes are separeted early in top-down method or merged into the groups late in bottom-up method. The minor difference between the two results is that digits 1 and 7 are considered to be similar in the top-down method, but not so in the bottom-up method.

The parent list for all 2K − 1 = 20 − 1 = 19 tree nodes is also shown in the figure below the tree structure generated by each of the two method. The 19 nodes are listed in the bottom row while their corresponding parent nodes are listed in the top row. For example, the parent node of leaf node 8 (for digit 7) is node 16 in the top-down method or node 12 in the bottom-up method. Based on this parent list, the tree struction can be automatically generated by the Matlab

function treeplot(p).

Once the binary tree structure is constructed, we can further construct a binary classifier at each of the K −1 = 10−1 = 9 non-leaf nodes based on a small set of

Figure 16.1 Binary Tree Classifier: Top-Down (left) and Bottom-Up (right)

features selected to saperate the two subgroups associated with the node. Such a sequence of binary classifications each based on a small set of features can be more effective and efficient than classifying all 10 class based on all features.

16.3 Binary Hierarchical Clustering

When the hierarchical method is applied to the clustering of a given dataset, the root node on top of the tree structure represents a super group containing all N data points in the dataset, while each of the leaf nodes is for one of these data point. A binary tree can be constructed in either top-down or bottom-up fashion. In the top-down method, the super group of all data points is recursively split into two subgroups in either depth-first or breadth-first fashion until reaching the bottom of the tree where each of the leaf nodes represents one of the N given data points. Alternatively, in the bottom-up method, starting from the leaf nodes each representing a single data sample, the two groups with minimum inter-group distance are merged to form a bigger group, and this process is further carried out recursively until reaching the top of the tree where the root node represents a super group containig all samples in the dataset.

The tree structure obtained by either method can then be truncated at certain level between the root node on top and the leaf nodes at the bottom to obtain a set of subgroups, each for one of the clusters, depending on the desired number and sizes of these clusters. Actually it is unnecessary to construct the complete tree first and then truncate it to get the clustering results. Instead, in either of the top-down or bottom-up methods, the recursion of splitting or merging can be terminated if certain conditions are met, such as the size of a group or the

Figure 16.2 Histogram of Between-Subgroup Distances

intra-cluster scatteredness is too large or too small. By doing so, we obtain a set of clusters without constructing the complete tree.

Similar to all unsupervised clustering methods such as K-means method (Section 15.1), hierarchical clustering also faces the difficulty of determining the proper K value for the number of clusters that best reflects the inherent structure of the dataset. Here specifically we need to determine the parameter values (size of clusters, intra-cluster scatteredness, etc.) for terminating the recursive merging or splitting at a proper level. If the tree structure is truncated at a level which is either too high or too low, the resulting K may be either too large or too small, causing the dataset to be either under or over divided, as illustrated in Fig. 16.4 for an example to be considered later.

One way to address the issue of under versus over subdivision is to generate the histogram of all between-class distances at the tree nodes, as shown in Fig. 16.2. In general, the larger distances on the right side of the histogram come from the subdivisions at the top levels by which the dataset is divided into subgroups, and the smaller distances on the left are from the subdivisions at the lower levels by which some clusters in the dataset may be unnecessarily and over subdivided. In other words, the larger and smaller distances may reflect respectively the inter-cluster and intra-cluster scatteredness respectively. Also we note that in the histogram, there exists a dense peak composed of many small intra-group distances, much more than the number of larger inter-cluster distances coarsely distributed towards the right end of the histogram.

It is therefore clear that recursive subdivision should be truncated at certain point along the histogram that separates the valid larger inter-cluster distances on the right from the invalid smaller intra-cluster distances on the left, as also shown in Fig. 16.2. Based on this method the tree structure can be properly truncated as shown in Fig. 16.4 for Example 16.2, where a dataset is under (left), properly (middle), or over (right) clustered.

The Matlab code for both the depth-first and breadth-first algorithms is listed below, containing the following variables:

• K: the number of clusters (leaf nodes) so far (initially zero)

• X: a d × N array containing N d-dimensional column vectors each for one of the samples.

• sid: sample indices for all samples (1, ⋯, N)

• kid: cluster labelings (1, ⋯, K) for all N samples, of which the nth component indicates the cluster sample x_{n} is assigned to.

• path: a list of paths (strings) for all N samples, only needed for visualization of the tree. For example, a sample with path ’010’ is assigned to the left, right, and left subgroups in three consecutive steps along a tree branch.

• mins: pre-specified minimum number of samples in a cluster.

• th: prespecified threshold value for within-cluster scatteredness.

The depth-first binary subdivision is based on based a stack (last-in-first-out), carried out recursively:

function DFdivide(sid)          % subdivide a node containg samples in sid
    global X K path kid th mins % global variables shared by other functions
    N=length(sid);              % number of samples
    Y=X(:,sid);                 % all samples at current node
    if cov2(Y)<th | N<mins      % terminate recursion (leaf node reached)
        K=K+1;                  % update number of clusters
        kid(sid)=K;             % assign cluster ID to all samples
        return
    end
% split group (sid) into left (lid) and right (rid) subgroups:
    [lid rid]=twoMeans(Y,sid);  % by either K-means (K=2)
%    [lid rid]=split(Y,sid);     % or splitting in principal direction
    for l=1:length(lid)         % for each sample on the left
        i=lid(l);               % index of each sample
        path{i}=[path{i},'0'];  % append 0 to path
    end
    for l=1:length(rid)         % for each sample on the right
        i=rid(l);               % index of each sample
        path{i}=[path{i},'1'];  % append 1 to path
    end
    % recursively subdivide the left and right nodes
    DFdivide(lid)
    DFdivide(rid)
end

Here cov2 is a function that calculates the scatteredness, trace of covariance of column vectors in dataset X:

function tcov=cov2(X)
[d,n]=size(X);
tcov=0;
if n>1
tcov=trace(cov(X'));
end
end

The breadth-first binary subdivision is based a queue (first-in-first-out), represented in the code by a dynamic 2-D array Q. Each tree node is represented by one row of Q containing indices of all samples assigned to the node (appended by zeros for all rows to have the same length).

function BFdivide(sid)
global X K path kid th mins
N=length(sid);                  % number of data samples
Q=[[];sid];                     % initial queue: a row of all samples
while size(Q,1)>0               % loop until the queue is empty
q=Q(1,:);                   % get the first row
sid=q(1:nnz(q));            % get sample IDs in the row
Y=X(:,sid);                 % samples to be subdivided by either
%        [lid rid]=twoMeans(Y,sid);  % by K-means method (K=2)
[lid rid]=split(Y,sid);     % or split along principal direction
for l=1:length(lid)         % record splitting history
    i=lid(l);
    path{i}=[path{i},'0'];  % paths of samples in left branch
end
for l=1:length(rid)
    i=rid(l);
    path{i}=[path{i},'1'];  % paths of samples in right branch
end
if cov2(X(:,lid))<th | length(lid)<mins    % a leaf node
    K=K+1;                  % update number of clusters
    kid(lid)=K;             % assign cluster ID to all samples
else                        % not a leaf node
    lid(end+1:N)=0;         % append zeros
    Q=[Q;lid];              % put left node back to queue
end
if cov2(X(:,rid))<th | length(rid)<mins  % same for left branch
    K=K+1;
    kid(rid)=K;
else
    rid(end+1:N)=0;
    Q=[Q; rid];
end

Figure 16.3 Tree Structures with 3, 7 and 15 Leaf Nodes Each for a Cluster

Figure 16.4 Under (left), Proper (middle), and Over (right) Division

        Q=Q(2:end,:);               % remove front row from queue
    end
end

Example 16.2 The top-down method is applied to the clustering of a set of 80 data points in a 4-D space, to generate a binary tree based on sequence of binary subdivision. The subdivision is terminated when either the number of samples or the intra-group scatteredness in a subgroup is smaller than a prespecified value. Based on three sets of such parameters, three tree structures are obtained as shown in Fig. 16.3, and the corresponding clustering results are shown in Fig. 16.4 (in 3-D space spanned by the first three pricipal components after KLT). We see that the dataset is best fit by the 7 clusters in the second case due to the properly selected parameters (th=0.01, mins=20), but it is either under divided (3 clusters in the first case) or over divided (15 clusters in the third case) due to improper parameters.

Example 16.3 The binary top-down splitting method is applied to the clustering analysis of a gene microarray containing a set of 99 genes (credit to the National Cancer Institute NIH, https://llmpp.nih.gov/lymphoma/data.shtml), in terms of their expression levels under seven different experimental conditions

Figure 16.5 Clustering of 99 genes

as shown in Fig. 16.5 where the expression levels are color coded. These genes are clustered into various subsets of similar genes groups in terms their expression levels under different conditions. The five panels in the figure show how the full dendrogram (top) can be truncated at various levels to form different numbers of subsets. For example, the bottom panel shows that these genes can be clustered into four groups, which can be further merged into two larger groups representing very different behaviors under the seven conditions. The same method is also carried out for the seven experimental conditions (instead of the 99 genes) so that they can be also be clustered into the subgroups each containing a set of similar conditions, as shown by the smaller dendrogram on the right side of the microarray data.

The method is further applied to the clustering of a much larger microarray dataset containing 13,411 genes (from the same source as above). The result is given in Fig. 16.6 showing only a small fragment of the complete dendrogram containing a small subset of the genes.

Problems

1. Develop your own code to implement a supervised hierarchical classifier by constructing a binary decision tree based on both top-down and bottom up methods, and then apply both of them to carry out classification of the handwritten digit dataset (10 classes with 256 features). Compare the two methods in terms of both accuracy (from the confusion matrices) and execution time.

2. Develop your own code for uusupervised hierarchical clustering based on both top-down and bottom-up methods, and apply them to the clustering of the hand-written digit dataset. Obtain the dendrograms and confusion matrices from the two methods and compare them in terms of accuracy and execution time.

Extra credit: apply your clustering code to a microarray dataset of reasonable size downloaded from: https://llmpp.nih.gov/lymphoma/data.shtml.

406  Hierarchical Classifiers

Figure 16.6 Clustering of 13,411 genes (only a small fragment shown)

Part V Neural Networks 17 Biologically Inspired Networks

In Part V, we consider the general learning method of artificial neural networks (ANN), a category of algorithms in machine learning that are inspired by the biological neural networks in the brain, composed of large number of neural cells called neurons. An ANN is designed to carry out either supervised learning such as regression as well as classification, or unsupervised learning such as clustering. The ANN method is covered in Part V, separate from all supervised and unsupervised learning methods in the previous parts, simply because the ANN formulates the learning problems in a rather unique way (as we will see below), quite different from all previous methods. However, both supervised and unsupervised ANN algorithms can also be discussed together with all previously considered algorithms, as in the case of many other textbooks.

Before discussing specific learning algorithms and the actual computations taking place in an ANN, we could treat it as a black box that mimic the behavior of a certain generic system, which generates a set of outputs as the response to a set of inputs, in the sense that the output of the ANN is similar to that of the system, if both are given the same input, although the inner workings of the ANN may be completely different from that of the system. In other words, the ANN can be used to model a system only in terms of the relationship between it input and output.

Mimicking the biological neural networks, an ANN is typically a hierarchical structure composed of multiple layers of neurons, also called nodes in the context of ANN, including the input layer, the hidden layers, and the output layer. These layers are organized hierarchically in the sense that each layer takes input from a lower layer and generates output to feed into a higher layer. As the computations taking place through the multilayers are in general nonlinear, an ANN can be used to model complex behaviors of some sophisticated systems such as visual signal processing and object recognition in the brain.

To achieve a specific learning goal, an ANN needs to be iteratively trained to learn the desired behavior from the given dataset, so that the weights representing the connectivities between the nodes can be progressively modified to generate the desired output given the input. In supervised learning, each sample x_{n} in X = [x_{1}, ⋯, x_{N}] of the training set is labeled by the corresponding y_{n} in Y = [y_{1}, ⋯, y_{N}], treated as the desired output to be compared with the actual output ŷ_{n} generated by the network, so that the weights can be modified in such a way that the future output will be closer to the desired y_{n}, i.e., r = y − ŷ will be reduced. Neural network algorithms formulated in this way is based on the general learning model described in Fig. 4.1 in Section 4.1 for regression and classification.

On the other hand, in unsupervised learning, the samples in the dataset X are not labeled, there exists no desired output for the ANN algorithm to aim at, and it needs to modify its weights on its own based on the data samples alone, to achieve the desired goal, such as clustering, similar to the K-means method.

17.1 Biological Inspiration

To understand how such a neural network algorithm works, we first consider some basic concepts in biological neural system. The human brain consists of 10^{11} neurons interconnected through about 10^{14} to 10^{15} synaptic junctions to form millions of neural networks for various functions. Hundreds specialized cortical areas are formed based on these networks for different information processing tasks.

Functionally, a neuron consists of the following three parts as shown in Fig. 17.1:

• The cell body or soma containing the nucleus of the cell;

• The dendrites that receive electrochemical stimulation (input impulses) from other neurons and propagate them to the cell body;

• The axon: that conducts the impulses (output) away from the cell body to other cells.

For the large number of neurons in a neural network to achieve a certain function collectively, they are connected by the axo-dendritic synapses where the dendrites of one neuron receive input stimuli transmitted through the axons of many other neuron. The computational model of such a neuron in the neural network, called a node, is also shown in Fig. 17.1, which receives input signals denoted by x_{1}, ⋯, x_{d} from d other neurons and becomes activated to different extents in terms of its action potential or simply activation, as a weighted sum of the inputs:

a = ∑_{j=1}^{d} w_{j}x_{j} + b = ∑_{j=0}^{d} w_{j}x_{j} = w^{T} x  (17.1)

We note that this weighted sum is exactly the same as the linear regression model in Eq. (5.1) in Section 5. However, here the model parameters take on different meanings: b is the offset or bias, and w_{j} is a weight representing the synaptic connectivity to the jth input node:

w_{j} \{ \begin{matrix}> 0 & excitatory input \\ < 0 & inhibitory input \\ = 0 & no connection \end{matrix}  (17.2)

Figure 17.1 Computational Model of Neurons

Same as in linear regression, here we have defined x_{0} = 1 and w_{0} = b, so that both the weight and pattern vectors are augmented to become d + 1 dimensional vectors x = [x_{0} = 1, x_{1}, ⋯, x_{d}]^{T} and w = [w_{0} = b, , w_{1}, ⋯, w_{d}]^{T}. Based on the activation level a, the neuron produces an output or response as a function of a:

y = g(a) = g (∑_{j=1}^{d} w_{j}x_{j} + b) = g(w^{T} x)  (17.3)

Here g(x), the activation function, typically takes one of the following forms:

• Liner function:

g(x) = x,  \frac{d g(x)}{dx} = 1  (17.4)

• Logistic function:

g(x, a) = \frac{1}{1 + e^{−ax}} = \frac{e^{ax}}{1 + e^{ax}} = \{ \begin{matrix}0 & x = −∞ \\ 1/2 & x = 0 \\ 1 & x = ∞ \end{matrix} , \frac{d g(x)}{dx} = \frac{a e^{−ax}}{(1 + e^{−ax})^{2}}

(17.5)

where parameter a controls the slope of the transition from 0 to 1 around x = 0. Specially

g(x, 0) = \frac{1}{2},  \frac{d g(x)}{dx} = 0  (17.6)
g(x, ∞) = u(x) = \{ \begin{matrix}0 & x < 0 \\ 1 & x > 0 \end{matrix} ,  \frac{d g(x)}{dx} = \{ \begin{matrix}0 & x ≠ 0 \\ ∞ & x = 0 \end{matrix} (17.7)

Figure 17.2 Activation Functions (left) and Their Derivatives (right) Logistic (top), Tanh (middle), and ReLU and Softplus (bottom)

• Tanh (hyperbolic tangent) function:

g(x, a) = \frac{2}{1 + e^{−ax}} −1 = \frac{e^{ax} − 1}{e^{ax} + 1} = \{ \begin{matrix}−1 & x = −∞ \\ 0 & x = 0 \\ 1 & x = ∞ \end{matrix} , \frac{d g(x)}{dx} = \frac{2a e^{−ax}}{(1 + e^{−ax})^{2}}

(17.8)

• Rectified linear unit (ReLU):

g(x) = max(0, x) = \{ \begin{matrix}0 & x < 0 \\ x & x > 0 \end{matrix} , \frac{d}{dx}g(x) = \{ \begin{matrix}0 & x < 0 \\ 1 & x > 0 \end{matrix} (17.9)

The soft version of ReLU is the softplus function of which the derivative is the logistic function:

g(x) = log(1 + e^{x}), \frac{d g(x)}{dx} = \frac{e^{x}}{1 + e^{x}} = \frac{1}{1 + e^{−x}}  (17.10)

These activation functions and their derivatives are shown in Fig. 17.2.

A neural network can be typically modeled by a hierarchical structure as shown in Fig. 17.3 containing two or more layers of neurons, called nodes in the ontext of artificial neural networks:

• The input layer: receives inputs from external sources;

• The output layer: generates output to the external world;

Figure 17.3 Artificial Neural Networks of Two (left) or Three (right) Layers

• The hidden layer(s): one or more layers between the input and output layers, not visible from outside the network.

A neural network can be trained according to certain mathematical rules, the learning rules or learning laws, by modifying its weights iteratively based on the inputs (and the desired outputs if the learning is supervised), so that the network can produce the desired output as the response to its input as the stimulus.

The method of ANN can be used for both supervised and unsupervised learning. For supervised learning, the goal is to model the relationship between the dependent variable y as the output of the network and the independent variable x as the input of the network by a function y = f(x, w), of which the parameter w for all weights in the network is to be estimated in the learning process based on the training set D = {X, y}. Such a general framework is suitable for both regression and classification. Neural networks can also be used for unsupervised learning based on observed data X without the labeling y.

Listed below are several learning paradigms for various neural network algorithms with different interpretations of the input and output of a neural network.

1. Pattern Associator

This is the most general form of neural network learning by which the associative relationship between two sets of patterns in X = [x_{1}, ⋯, x_{N}] and Y = [y_{1}, ⋯, y_{N}] is learned and stored.

• Training: A set of N pairs of patterns {(x_{n}, y_{n}), n = 1, ⋯, N} is presented to the network which then learns to establish the associative relationship between two sets of patterns:

f : x ∈ R^{d} ⟹ y ∈ R^{m}  (17.11)

• Testing: When an unlabeled pattern x in a pair is presented as the input,

the network produces an output pattern y associated to the output. Human memory is associative in the sense that given one pattern, some as-

sociated pattern(s) may be produced. Examples include: (Evolution, Darwin), (Einstein, E = mc^{2}), (food, sounding bell, salivation).

2. Auto-associator

As a special pattern associator, auto-associator associates a pre-stored pattern to an incomplete or noisy version of the pattern.

• Training: A set of patterns {x_{1}, ⋯, x_{N}} is presented to the network for it to learn and remember, i.e., the patterns are stored in the network.

• Testing: When an incomplete or noisy version of one of the patterns stored in the network is presented as the input to the network, the original pattern is retrieved by the network as the output.

3. Regression

This is another special kind of pattern associator which takes a vector input x ∈ R^{d} and produces a real value y ∈ R as a multivariable function y = f(x) at its only output node.

• Training: trained by a set of observed data samples, the independent vectors and their corresponding function values {(x_{n}, y_{n}), n = 1, ⋯, N}, the network is model the function.

• Testing: given any vector x, the output value produced by the single output

node is an estimated function value y = f(x).

4. Classification

This is a variation of the pattern associator of which the output patterns are a set of categorical symbols representing different classes {C_{1}, ⋯, C_{K}}, i.e., each input pattern is classified by the network into one of the classes

f : x ∈ R^{d} ⟹ y ∈ {C_{1}, ⋯, C_{K}}  (17.12)

5. Regularity Detector

This is an unsupervised learning process. The network discovers automatically the regularity in the inputs so that similar patterns are automatically detected and grouped together in the same cluster or class.

17.2 Hebbian Learning

Donald Hebb (1949) speculated that

“When neuron A repeatedly and persistently takes part in exciting neuron B, the synaptic connection from A to B will be strengthened.”

In other words, simultaneous activation of neurons leads to pronounced increases in synaptic strength between them, or “neurons that fire together wire together; neurons that fire out of sync, fail to link”.

For example, the well known classical conditioning (Pavlov, 1927) could be explained by Hebbian learning. Consider the following three patterns:

• Unconditioned stimulus: sight of food F

• Conditioned stimulus: sound of bell B

• Response: secretion of saliva S The unconditioned response is: F → S. Due to the repeated and persistent conditioning process F ∩ B → S, the synaptic connections between patterns B and S are strengthened as both are repeatedly excited simultaneously, i.e., the two patterns become associated, resulting the conditioned response B → S.

Based on this theory of Hebbian learning, the Hebbian network can be considered as a supervised learning method that learns to establish the associative relationship between any pair of two patterns x_{n} and y_{n} in the given dataset {x_{1}, ⋯, x_{N}} and {y_{1}, ⋯, y_{N}}, considered as the training set. This is a 2-layer network with d nodes in the input layer to receive an input pattern x = [x_{1}, ⋯, x_{d}]^{T} and m nodes of the output layer to produce an output y = [y_{1}, ⋯, y_{m}]^{T}. Each output node is fully connected to all d input nodes through its weights and generates an output:

y_{i} = ∑_{j=1}^{d} w_{ij}x_{j} = w_{i}^{T} x (i = 1, ⋯, m)  (17.13)

where w_{i} = [w_{i1}, ⋯, w_{id}]^{T} is the weight vector for the ith output node. These m equations can be written in matrix form as

y = [ y_{1} ; ⋮ ; y_{m} ] = [ w_{1}^{T} ; ⋮ ; w_{m}^{T} ] [ x_{1} ; ⋮ ; x_{d} ] = Wx  (17.14)

where W = [w_{1}, ⋯, w_{m}]^{T} is an m × d matrix.

The Hebbian learning rule is inspired by Hebb’s theory, i.e., when both neurons x_{j} and y_{i} are activated, the synaptic connectivity, here the weight w_{ij}, between them is enhanced:

w_{ij}^{new} = w_{ij}^{old} + η x_{j}y_{i} (i = 1, ⋯, m, j = 1, ⋯, d)  (17.15)

or in matrix form:

W^{new} = W^{old} + η yx^{T}  (17.16)

Here η is the learning rate, a parameter that controls how quickly the weights get modified.

As in all supervised learning, the Hebbian network is first trained and then used for association.

• Training:

For simplicity, we assume all weights are initialized to zero w_{ij} = 0 (i = 1, ⋯, m, j = 1, ⋯, d), and then train the network to find all weights based on all N pattern pairs in the dataset {(x_{n}, y_{n}), n = 1, ⋯, N} based on the learning law:

w_{ij} = ∑_{n=1}^{N} x_{j}^{(n)}y_{i}^{(n)} (i = 1, ⋯, m, j = 1, ⋯, d)  (17.17)

or in matrix form, the weight matrix is the sum of the outer-products of all N pairs of patterns:

W_{m×d} = ∑_{n=1}^{N} y_{n}x_{n}^{T} = ∑_{n=1}^{N} [ y_{1}^{(n)} ; ⋮ ; y_{m}^{(n)} ] [x_{1}^{(n)}, ⋯, x_{d}^{(n)}]  (17.18)

• Association:

When one of the patterns x_{l} is presented to the network, it produces an output:

y = Wx_{l}  (17.19)

which, as shown below, is the same as the associated pattern y_{l}, under the following conditions:

– The N patterns {x_{1}, ⋯, x_{N}} are treated as zero-mean random vectors, and they are assumed to be independent of each other with zero covariance and therefore zero correlation coefficient between any x_{i} and x_{j}:

r_{ij} = \frac{σ_{ij}^{2}}{σ_{i}σ_{j}} = \frac{∑_{k=1}^{d} x_{ki}x_{kj}}{\sqrt{∑_{k=1}^{d} x_{ki}^{2}}\sqrt{∑_{k=1}^{d} x_{kj}^{2}}} = \frac{x_{i}^{T} x_{j}}{||x_{i}|| ||x_{j}||} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}

(17.20)

If we further assume all patterns are normalized with ||x_{n}|| = 1, then we have x_{i}^{T} x_{j} = δ_{ij}

– The number of input nodes is greater than the number of pattern pairs: d ≥ N, i.e., the capacity of the network is large enough for representing N different patterns (as there can be no more than d orthogonal vectors in a d-dimensional space).

Under these assumed conditions, the output of the network as its response to input x_{l} is

y = Wx_{l} = (∑_{n=1}^{N} y_{n}x_{n}^{T} ) x_{l} = y_{l}(x_{l}^{T} x_{l}) + ∑_{n≠l} y_{n}(x_{n}^{T} x_{l}) = y_{l} (17.21)

as x_{n}^{T} x_{l} = 0 for all other terms with n ≠ l. We see that a one-to-one correspondence relationship between x_{n} and y_{n} has been established for all n = 1, ⋯, N.

In non-ideal cases where the conditions above are not strickly satisfied, e.g., the patterns in the dataset are not completely independent, the summation term is non-zero and there will be a non-zero error y − y_{l} ≠ 0. However, the Hebbian network trained in this way may still behave as an associator that mimic approximately the general associative relationship between x_{l} and y_{l}.

Figure 17.4 The Hopfield Network

17.3 Hopfield Network

The Hopfield network is a recurrent neural network (RNN) that allows the previous outputs of the network to be fed back and used as inputs again (with time delay) as an iterative process, so that it can learn to respond to inputs as a temporal sequence, such as a speech signal.

The structure of a Hopfield network is illustrated in Fig. 17.4. The learning of the Hopfield network is based on the Hebbian learning rule, and as a supervised method, it is trained on a dataset X = [x_{1}, ⋯, x_{K}] of K data samples each composed of d binary components x = [x_{1}, ⋯, x_{d}]^{T} with x_{i} ∈ {−1, 1}, (i = 1, ⋯, d), representing one of K different patterns of interest, used as the inputs to the d nodes of network. Once the network is completely trained, the weight matrix W = [w_{1}, ⋯, w_{d}] of the network is obtained in which the K patterns are stored. Given an input x, an iterative computation is carried out until convergence, when one of the pre-stored pattern that most closely resemble the current input x is produced as the output. The Hopfield network can therefore be considered as an auto-associator (a content addressable memory), by which a pre-stored pattern can be retrieved based on its association to the input pattern established by training process of the network.

• Training:

The training process is essentially the same as the Hebbian learning, except here the two associated patterns in each pair are the same (selfassociation). The weight matrix of the network is obtained as the sum of the outer-products of the K patterns to be stored:

W_{d×d} = ∑_{k=1}^{K} x_{k}x_{k}^{T} = ∑_{k=1}^{K} [ x_{1}^{(k)} ; ⋮ ; x_{d}^{(k)} ] [x_{1}^{(k)}, ⋯, x_{d}^{(k)}]  (17.22)

The weight connecting node i and node j is defined as

w_{ij} = ∑_{k=1}^{K} x_{i}^{(k)}x_{j}^{(k)} = w_{ji}  (17.23)

We assume no self-connection exists w_{ii} = 0 (i = 1, ⋯, d)

• Autoassociation:

Once the weight matrix W is obtained by the training process, the network can be used as a self-associator. When an input pattern x is presented to the network, the outputs of the network are updated iteratively and asynchronously, one randomly selected node at a time:

x_{i}^{(n+1)} = sign (∑_{j=1}^{d} w_{ij}x_{j}^{(n)}) = \{ \begin{matrix}+1, & if ∑_{j=1}^{d} w_{ij}x_{j}^{(n)} ≥ 0 \\ −1, & if ∑_{j=1}^{d} w_{ij}x_{j}^{(n)} < 0 \end{matrix} (17.24)

where x_{i}^{(n)} and x_{i}^{(n+1)} are the output x_{i} of the ith node before and after the nth iteration, respectively. As shown below, this iteration will always converge to one of the K pre-stored patterns.

We first define the Energy function of any two nodes x_{i} and x_{j} of x as

e_{ij} = −w_{ij}x_{i}x_{j}  (17.25)

and the total energy of all d nodes in the network as the sum of all pair-wise energies:

E(x) = \frac{1}{2} ∑_{i=1}^{d} ∑_{j=1}^{d} e_{ij} = −\frac{1}{2} ∑_{i=1}^{d} ∑_{j=1}^{d} w_{ij}x_{i}x_{j} = −\frac{1}{2}x^{T} Wx  (17.26)

The interaction between these two nodes is summarized below:

\begin{aligned}& x_{j} & x_{i} & w_{ij} > 0 & w_{ij} < 0 \\ 1 & −1 & −1 & e_{ij} < 0 & e_{ij} > 0 \\ 2 & −1 & 1 & e_{ij} > 0 & e_{ij} < 0 \\ 3 & 1 & −1 & e_{ij} > 0 & e_{ij} < 0 \\ 4 & 1 & 1 & e_{ij} < 0 & e_{ij} > 0 \end{aligned}  (17.27)

We make two observations.

• When the two nodes i and j reinforce each other’s state, e_{ij} < 0:

– If w_{ij} > 0 (in cases 1 and 4), x_{j} = ∓1 tends to keep x_{i} to stay at the same state ∓1 in the iteration.

– If w_{ij} < 0 (in cases 2 and 3), x_{j} = ∓1 tends to keep x_{i} to stay at the

same state ±1.

• When the two nodes i and j change each other’s state, e_{ij} > 0:

– If w_{ij} < 0 (in cases 1 and 4), x_{j} = ∓1 tends to reverse x_{i} from its previous state ∓1 to ±1.

– If w_{ij} > 0 (in cases 2 and 3), x_{j} = ∓1 tends to reverse x_{i} from its previous state ±1 to ∓1.

We note that low energy e_{ij} corresponds to a stable interaction between x_{i} and x_{j}, i.e., they tend to remain unchanged, and high energy corresponds to an unstable interaction, i.e., they tend to change their states. As the result, low total E(x) corresponds to more stable condition of the network, while high E(x) corresponds to less stable condition.

We further show that the total energy E(x) always decreases whenever the state of any node changes. Assume x_{k} has just been changed, i.e.,x_{k}^{(n+1)} ≠ x_{k}^{(n)} (x_{k} = ±1 but x_{k}^{(n+1)} = ∓1), while all others remain the same x_{l≠k}^{(n+1)} = x_{l≠k}^{(n)}. The energy before x_{k} changes state is

E^{(n)}(x) = −\frac{1}{2} [∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}^{(n)}x_{j}^{(n)} + ∑_{i} w_{ik}x_{i}^{(n)}x_{k}^{(n)} + ∑_{j} w_{kj}x_{k}^{(n)}x_{j}^{(n)}]

= −\frac{1}{2} ∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}x_{j} − ∑_{i} w_{ik}x_{i}^{(n)}x_{k}^{(n)}

and the energy after x_{k} changes state is

E^{(n+1)}(x) = −\frac{1}{2} ∑_{i≠k} ∑_{j≠k} w_{ij}x_{i}x_{j} − ∑_{i} w_{ik}x_{i}^{(n+1)}x_{k}^{(n+1)}  (17.28)

The energy difference is

△E = (x_{k}^{(n)} − x_{k}^{(n+1)}) ∑_{i} w_{ik}x_{i}  (17.29)

Consider two cases:

• Case 1: if x_{k}^{(n)} = −1, but ∑_{i} w_{ik}x_{i} ≥ 0 and x_{k}^{(n+1)} = 1, we have (x_{k}^{(n)} − x_{k}^{(n+1)}) = −2 ≤ 0 and △E ≤ 0.

• Case 2: if x_{k}^{(n)} = 1, but ∑_{i} w_{ik}x_{i} < 0 and x_{k}^{(n+1)} = −1, we have (x_{k}^{(n)} − x_{k}^{(n+1)}) = 2 ≥ 0 and △E ≤ 0.

As in either case, △E(x) ≤ 0 is always true throughout the iteration, the energy E(x) will eventually reach one of the minima of the energy landscape shown in Fig. 17.5, i.e., the iteration will always converge.

We further show that each of the K pre-stored patterns corresponds to one of the minima of the energy function. The energy function E(x) corresponding to

Figure 17.5 Energy Landscape

any x can be written as

E(x) = −\frac{1}{2}x^{T} Wx = −\frac{1}{2}x^{T} [∑_{k=1}^{K} x_{k}x_{k}^{T} ] x = −\frac{1}{2} ∑_{k=1}^{K} x^{T} x_{k}x_{k}^{T} x = −\frac{1}{2} ∑_{k=1}^{K}(x^{T} x_{k})^{2}

(17.30)

If x is different from (ideally orthogonal to) any of the K stored patterns, all terms in the summation will be small (ideally zero). But if x is similar to (ideally the same as) any one of the stored patterns, their inner product will be large (ideally reach maximum), causing the total energy to be minimized to reach one of the minima. In other words, the patterns stored in the network correspond to the local minima of the energy function. i.e., these patterns become attractors.

Note that it is possible to have other local minima, called spurious states, which do not represent any of the stored patterns, i.e., the associative memory is not perfect.

18 Perceptron-Based Networks

18.1 Perceptron

The perceptron network (Frank. Rosenblatt, 1958) is a two-layer network containing an input layer of d nodes and an output layer of m output node. As a supervised method, it is trained by a dataset X = [x_{1}, ⋯, x_{N}], of which each sample x = [x_{1}, ⋯, x_{d}]^{T} is labeled by the corresponding component in y = [y_{1}, ⋯, y_{N}]^{T}, indicating to which one of K categorical classes x_{n} belongs. The goal of the training process is to determine the weights w = [w_{1}, ⋯, w_{d}]^{T} and bias b so that the output ŷ_{n} = ∑_{i=1}^{d} w_{i}x_{n} + b corresponding to input x_{n} matches its labeling y_{n} for all N samples in the training set in some optimal way. When the perceptron network is fully trained, it can be used to classify any unlabeled sample x into one of the K classes.

We first consider the special case where the output layer has only m = 1 node, and the perceptron is a binary classifier, same as the method of linear regression, and also the initial setup of the support vector machine (SVM), in the sense that all such methods are based on the linear equation f(x) = w^{T} x + b = 0, representing a hyperplane by which the d-dimensional feature space is partitioned into two regions corresponding to two classes C_{+} and C_{−}. When the parameters w and b are determined in the training process, any unlabeled x is classified into either of the two classes depending on whether the function value ŷ = f(x) is greater or smaller than zero:

If f(x) = w^{T} x + b \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then ŷ = sign(f(x)) = \{ \begin{matrix}1 & for x ∈ C_{+} \\ −1 & for x ∈ C_{−} \end{matrix}

(18.1)

If we divide the equation f(x) = w^{T} x + b = 0 by ||w||, we get

p_{w}(x) = \frac{w^{T} x}{||w||} = −\frac{b}{||w||} = b′  (18.2)

where p_{w}(x) = w^{T} x/||w|| is the projection of x onto the normal direction w of the partitioning hyperplane, and b′ = −b/||w|| is the vector from the origin to the hyperplane (|b′| is the distance of the hyperplane to the origin). Now the classification above can be rewritten as

If p_{w}(x) \{ \begin{matrix}> b′ \\ < b′ \end{matrix} , then x ∈ \{ \begin{matrix}C_{+} \\ C_{−} \end{matrix}  (18.3)

We see that in all binary classification methods, an unlabeled sample x is classified into either of the two classes based on the projection p_{w}(x) of x onto w, which is either greater or smaller than the bias b′, indicating x is on either the positive or negative side of the plane.

In all these binary classification methods the parameters w and b need to be determined based on training set X labeled by y, but they do so differently. While in least squares method and support vector machine these parameters are obtained by the least squared method and quadratic programming respectively, here in the perceptron network these parameters are obtained by iteratively modifying some randomly initialized values to gradually reduce the error or residual, the difference between actual output ŷ_{n} = f(x_{n}) and the ground truth labeling y_{n}, denoted by δ = y_{n} − ŷ_{n}, for all training samples in the dataset. This method is called the δ-rule.

We now consider specifically the training algorithm of the perceptron network as a binary classifier. As always, we redefine the data vector as x = [x_{0} = 1, x_{1}, ⋯, x_{n}]^{T} and the weight vector as w = [w_{0} = b, w_{1}, ⋯, w_{n}]^{T} so that the decision function can be more conveniently written as an inner product f(x) = w^{T} x without the additional bias term.

The randomly initialized weight vector w is iteratively updated based on the following error driven δ-rule:

w^{new} = w^{old} + η(y − ŷ) x = w^{old} + η δx  (18.4)

where η > 0 is the step size but here called the learning rate, which is assumed to be 1 in the following for simplicity. We can show that by the iteration above, w is modified in such a way that the error δ is always reduced.

When a training sample x labeled by y ± 1 is presented to the nodes of the input layer of the perceptron, its output ŷ = sign(f(x)) may or may not match the the label y, as shown in the table:

y  f(x) = w^{T} x, ŷ = sign(f(x)) δ = y − ŷ
1 y = 1 f(x) > 0, ŷ = 1  δ = 0
2 y = −1 f(x) > 0, ŷ = 1  δ = −2
3 y = 1 f(x) < 0, ŷ = −1  δ = 2
4 y = −1 f(x) < 0, ŷ = −1  δ = 0

(18.5)

In both the first and last cases, yf(x) = 1 > 0, the error is δ = y − ŷ = 0, and the weight vector w^{new} = w^{old} + δx = w^{old} is not modified. But in cases 2 and 3, yf(x) = −1 < 0, the error is δ = y − ŷ ≠ 0, the weight vector w is to be modified in either of the following two ways:

• In case 2, y = −1, but ŷ = 1, then δ = −2 and y f(x) < 0, the weight vector is update by

w^{new} = w^{old} − 2x  (18.6)

When the same x is presented to the network again in the future, the function is

f(x) = x^{T} w^{new} = x^{T} w^{old} − η2x^{T} x < x^{T} w^{old}  (18.7)

which is smaller than its previous value and the output ŷ is more likely to be the same as the desired y = −1.

• In case 3, y = 1, but ŷ = −1, then δ = 2 and y f(x) < 0, the weight vector is

update by

w^{new} = w^{old} + 2x  (18.8)

When the same x is presented again, the function is

f(x) = x^{T} w^{new} = x^{T} x^{old} + η2x^{T} x > x^{T} w^{old}  (18.9)

which is greater than its previous value and the output ŷ is more likely to be the same as the desired y = 1.

The update equations in Eq. (18.6) for y = −1 and Eq. (18.8) for y = −1 can be combined to become

w^{new} = w^{old} + yx  (18.10)

where the scaling constant 2 is dropped as it can be absorbed into the learning rate if we let η = 1/2. Now the learning rule can be rewritten as:

If y f(x) \{ \begin{matrix}> 0 \\ < 0 \end{matrix} then \{ \begin{matrix}w^{new} = w^{old} \\ w^{new} = w^{old} + y x \end{matrix}  (18.11)

In summary, the learning law guarantees that the weight vector w is modified in such way that the performance of the network is always improved with reduced error δ = y−ŷ. If C_{+} and C_{−} are linearly separable, then a perceptron will always produce w in finite number of iterations to separate them.

This binary classifier with m = 1 output node can be generalized to multiclass classifier with m > 1 output nodes, and each of the m weight vectors in W = [w_{1}, ⋯, w_{m}] is modified by the same learning process considered above. The K > 2 classes can be encoded by the m outputs y_{i} ∈ {−1, 1}, (i = 1, ⋯, m) in two different ways. If the one-hot method is used, the m binary output can encode K = m classes, i.e., the kth class is represented by an m-bit output with y_{k} = 1 while all others y_{l} = −1 for l ≠ k. Alternatively, if binary encoding is used, the m outputs can encode as many as K = 2^{m} classes. For example, K = 4 classes can be labeled by 4 binary vector of either 4 or 2 bits:

binary output y_{0} y_{1} y_{2} y_{3}

y_{1}  1 −1 −1 −1
y_{2}  −1 1 −1 −1
y_{3}  −1 −1 1 −1
y_{4}  −1 −1 −1 1

or

binary output y_{0} y_{1} y_{2} y_{3}
y_{1}  −1 −1 1 1
y_{2}  −1 1 −1 1

(18.12)

The outputs of the m output nodes form an m-dimensional binary vector ŷ = [ŷ_{1}, ⋯, ŷ_{m}]^{T}, which is to be compared with the labeling y for the current input x with error δ = ||y − ŷ||. When the training is complete, any unlabeled input x is classified to one of the K classes with a matching label to the perceptron’s output. In the case of one-hot encoding, it is possible for the binary output y to have more than one hot node (e.g., ŷ = [−1 1 − 1 1]^{T}) not match any of the K one-hot encoded classes. In this case, the input x can be classified to the class corresponding to the node with the greatest output value f(x) = w^{T} x.

The Matlab code for the essential part of the algorithm is listed below. X = [x_{1}, ⋯, x_{N}] is the dataset contains N training samples, each labeled by one of the components of y = [y_{1}, ⋯, y_{n}] as its class identity. W is a (d + 1) × m matrix containing m d + 1 dimensional weight vectors each for one of the m output nodes.

[X Y]=DataOneHot;      % get data
K=length(unique(Y','rows')) % number of classes
X=[ones(1,N); X];      % data augmentation
[d N]=size(X);         % numbers of dimensions and samples
m=size(Y,1);           % number of output nodes
W=2*rand(d,m)-1;       % random initialization of weights
eta=1;
nt=10^4;               % maximum number of iteration
for it=1:nt
    n=randi([1 N]);    % random index
    x=X(:,n);          % pick a training sample x
    y=Y(:,n);          % label of x
    yhat=sign(W'*x);   % binary output
    delta=y-yhat;      % error = desired - actual
    for i=1:m          % for each of m output nodes
        W(:,i)=W(:,i)+eta*delta(i)*x;
                       % update weights for ith output node
    end
    if ~mod(it,N)      % test for every epoch
        er=test(X,Y,W);
        if er<10^(-9)
            break
        end
    end
end

This is the function that tests the training set based on estimated weight vectors in W:

function er=test(X,Y,W)  % test based on estimated W,
    [d N]=size(X);

Ne=0;                % number of misclassifications
for n=1:N
    x=X(:,n);
    yhat=sign(W'*x);
    delta=Y(:,n)-yhat;
    if any(delta)    % misclassification in some output nodes
        Ne=Ne+1;     % update number of misclassifications
    end
end
er=Ne/N;             % error percentage
end

This is the code that generates the training set labeled by either one-hot or binary encoding method:

function [X,Y]=Data
d=3;  K=8;
onehot=input('One-hot (1) or binary encoding (0): ');
Means=[-1 -1 -1 -1 1 1 1 1;
       -1 -1 1 1 -1 -1 1 1;
       -1 1 -1 1 -1 1 -1 1];
Nk=50*ones(1,K);
N=sum(Nk);          % total number of samples
X=[];
Y=[];
s=0.2;              % variance of noise
for k=1:K           % for each of K classes
    Xk=Means(:,k)+s*randn(d,Nk(k));
    if onehot
        Yk=-ones(K,Nk(k));
        Yk(k,:)=1;
    else            % binary encoding
        dy=ceil(log2(K));
        y=2*de2bi(k-1,dy)-1;
        Yk=repmat(y',1,Nk(k));
     end
    X=[X Xk];
    Y=[Y Yk];
end
Visualize(X,Y)
end

Example 18.1 Fig. 18.1 shows the classification results of a perceptron network with d = 2 input nodes for a 2-D dataset and m = 2 output nodes that encode

Figure 18.1 Classification by Perceptron (2-D)

Figure 18.2 Classification by Perceptron (3-D)

2^{2} = 4 classes. The training set contains N = 400 data samples including 100 samples for each of the four classes labeled by y = [y_{1}, y_{2}]. The two weight vectors w_{1} and w_{2} are initialized randomly. During the training iteration, the weight vectors are modified if δ = y −ŷ ≠ 0, otherwise they remain the same. After nine such modifications, the four classes are completely separated by the two straight lines normal to the two weight vectors. The first panel shows the initial stage, while the subsequent panels show how the weight vectors are modified each time when δ ≠ 0. The darker and bigger dots represent the samples presented to the network when one of the weight vectors is modified.

Fig. 18.2 shows the classification results of a perceptron of d = 3 input nodes and m = 3 output nodes that encode 2^{3} = 8 classes. After 35 modifications the weight vectors, the perceptron is completely trained to classify all eight classes correctly. The first panel shows the initial stage, the following three panels show the weight vectors, w_{1}, w_{2} and w_{3}, also the normal vectors of the decision planes, from some three different viewing angles. We see that the eight classes are indeed separated by the three planes normal to the weight vectors.

The perceptron algorithm as a binary classifier is based on the assumption that the two classes are linearly separable. This limitation can be removed by the kernel method, if the algorithm is modified in such a way that all data samples appear only in the form of an inner product. Consider first the training process in which the N training samples are repeatedly presented to the network, and the weight vector is modified to become w^{new} = w^{old} + y_{n}x_{n} whenever a sample x_{n} labeled by y_{n} is misclassified with δ = y_{n} − ŷ_{n} ≠ 0. If the weight vector is initialized to zero w = 0, then the weight vector by the updating rule in Eq. (18.10) can be written as a linear combination of the N training samples:

w = ∑_{n=1}^{N} α_{n}y_{n}x_{n}  (18.13)

where α_{n} is the number of times sample x_{n} labeled by y_{n} is misclassified. Upon receiving a new training sample x_{l} labeled by y_{l}, we have

f(x_{l}) = w^{T} x_{l} = ∑_{n=1}^{N} α_{n}y_{n}x_{n}^{T} x_{l}, and ŷ_{l} = sign(f(x_{l}))  (18.14)

and the weight vector w is updated by the delta-rule:

If δ = y_{l} − ŷ_{l} ≠ 0
then w^{new} = w^{old} + y_{l} x_{l} = ∑_{n=1}^{N} α_{n}y_{n}x_{n} + y_{l} x_{l}
i.e. α_{l}^{new} = α_{l}^{old} + 1  (18.15)

We see that only {α_{1}, ⋯, α_{N}} need to be updated during the training process, while the weight vector w in Eq. (18.13) no longer needs to be explicitly calculated. Once the training process is complete, any unlabeled x can be classified into either of the two classes based on {α_{1}, ⋯, α_{N}}:

If w^{T} x = ∑_{n=1}^{N} α_{n}y_{n}x_{n}^{T} x \{ \begin{matrix}> 0 \\ < 0 \end{matrix} , then x ∈ \{ \begin{matrix}C_{−} \\ C_{+} \end{matrix}  (18.16)

As all data samples appear in the form of inner product in both the training and testing phase, the kernel method can be applied to replace the inner product x_{n}^{T} x_{m} by a kernel function k(x_{n}, x_{m}). Also, the discussion above for m = 1 output nodes can be generalized to m > 1 output nodes.

Here is the Matlab code segment for the most essential parts of the kernel perceptron algorithm:

[X Y]=Data;                 % get dataset
[d N]=size(X);              % numbers of dimensions and samples
X=[ones(1,N); X];           % augmented data
m=size(Y,1);                % number of output nodes
A=zeros(m,N);               % initial alphas for all nodes and samples
K=Kernel(X,X);              % get kernel matrix of all N samples
for it=1:nt
    n=randi([1 N]);         % random index

x=X(:,n);               % pick a training sample
y=Y(:,n);               % and its label
yhat=sign((A.*Y)*K(:,n));   % get yhat
delta=Y(:,n)-yhat;      % error = desired - actual
for i=1:m               % for each output node
   if delta(i)~=0       % if a misclassification
       A(i,n)=A(i,n)+1; % update the corresponding alpha
   end
end
if ~mod(it,N)           % test for every epoch
    er=test(X,Y,A);     % percentage of misclassification
    if er<10^(-9)
        break
    end
end
end
function er=test(X,Y,A)     % function for testing
[d N]=size(X);          % dimension, number of samples
m=size(Y,1);
Ne=0;                   % number of misclassifications
for n=1:N               % for all N training samples
    x=X(:,n);           % get the nth sample
    y=Y(:,n);           % and its label
    f=(A.*Y)*Kernel(X,x); % f(x)
    yhat=sign(f);       % yhat=sign(f(x))
    if norm(y-yhat)~=0  % misclassification at some output nodes
        Ne=Ne+1;        % increase number of misclassification
    end
end
er=Ne/N;                % percentage of misclassification
end

Example 18.2 The kernel perceptron is applied to the classification of the iris dataset with m = 3 one-hot output nodes for the three classes, based on the radial basis kernel. The network is trained on half of the data and tested on the other half. The confusion matrix is shown below and the error rate is 14.67%.

[ \begin{matrix}22 & 6 & 0 \\ 0 & 18 & 5 \\ 0 & 0 & 24 \end{matrix} ]

Example 18.3 The kernel perceptron is applied to the handwritten digit dataset with m = 10 one-hot output nodes, based on the radial basis kernel. The kernel perceptron is trained on half of the data, and then tested one both the training half and the testing half of the data, and the the confusion matrices are shown below, and the corresponding error rates are zero when tested on the training half of the data and 0.11 when tested on the testing half.

[ \begin{matrix}116 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 108 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 116 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 113 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 122 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 103 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 108 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 109 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 116 \end{matrix} ]

[ \begin{matrix}104 & 7 & 0 & 0 & 0 & 1 & 2 & 1 & 0 & 1 \\ 0 & 103 & 0 & 0 & 0 & 0 & 3 & 1 & 0 & 1 \\ 1 & 0 & 103 & 0 & 0 & 0 & 1 & 1 & 2 & 0 \\ 0 & 0 & 8 & 102 & 0 & 0 & 0 & 2 & 1 & 3 \\ 1 & 1 & 0 & 0 & 103 & 2 & 0 & 0 & 4 & 2 \\ 0 & 0 & 0 & 2 & 0 & 93 & 24 & 0 & 3 & 1 \\ 0 & 1 & 1 & 0 & 0 & 0 & 95 & 4 & 1 & 1 \\ 0 & 0 & 1 & 0 & 0 & 0 & 0 & 104 & 0 & 3 \\ 1 & 0 & 1 & 1 & 3 & 2 & 1 & 3 & 93 & 4 \\ 1 & 2 & 0 & 1 & 7 & 0 & 0 & 3 & 6 & 96 \end{matrix} ]

The capability of the perceptron algorithm is limited in the sense that there is only one level of learning taking place between the output and input layers. This situation can be much improved by including additional learning layers between the input and output layers to form a multi-layer network, such as the back propagation which has been most widely used as a supervised learning algorithm, to be discussed in the next section.

18.2 Back Propagation

The back propagation network (BP network or BPN) is a supervised learning algorithm that finds a wide variety of applications in practice. In the most general sense, a BPN can be used as an associator that learns the relationship between two sets of patterns represented in vector forms. Specifically in classification, similar to the perceptron network, the BPN as a classifier is based on the training set X = [x_{1}, ⋯, x_{N}], of which each pattern x_{n} is associated with the corresponding pattern, an m-dimensional vector y_{n} in Y = [y_{1}, ⋯, y_{N}], as its class identity labeling, indicating to to which of the K classes {C_{1}, ⋯, C_{K}} pattern x_{i} belongs.

Different from the perceptron in which there is only one level of learning taking place between the output and input layers in terms of the weights, a BPN is a multi-layer (three or more) hierarchical structure composed of the input layer, one or more hidden layers, and the output layer, in which learning takes place in multiple levels between two consecutive layers. Consequently, a BPN can therefore be more flexible and powerful than the two-layer perceptron network, and more capable to learn more complicated and patterns.

In the following, before we consider the general BPN containing multiple hidden layers, we will first derive the back propagation algorithm for the simplest BPN with only one hidden layer in between the input and output layers, as shown in Fig. 18.3. We note that there are d+1 input nodes for input x = [x_{1}, ⋯, x_{d}]^{T} and a constant x_{0} = 1, and l + 1 hidden nodes that generate z = [z_{1}, ⋯, z_{l}]^{T} and a constant z_{0} = 1, while the output layer simply contain m nodes for the output y = [y_{1}, ⋯, y_{m}]^{T}.

Each node in the hidden (excluding z_{0} = 1) and output layers is fully connected to all nodes in the previous layer. When one of the N training patterns x_{n} is presented to the input layer of the BPN, an m-D vector ŷ_{n} = f(x_{n}, W^{h}, W^{o}) is produced at the output layer as the corresponding response to the input x_{n}. Here W^{h} = [w_{1}^{h}, ⋯, w_{l}^{h}] (a (d + 1) × l matrix) and W^{o} = [w_{1}^{o}, ⋯, w_{m}^{o}] (a (l + 1) × m matrix) are the model parameters containing the augmented weight vectors for both the hidden and output layers, to be determined in the training process based on the training set X and corresponding labeling Y, so that its output ŷ_{n} as a function of the current input x_{n} matches the desired output or the target, the labeling y_{n} as closely as possible. Once the BPN is fully trained, any unlabeled pattern x can be classified into one of the K classes corresponding to the minimum δ = ||y − ŷ||. Note that different from the perceptron network, the output of the m output nodes take real values in general, although they may still be binary (such as one-hot or binary encoding for class labeling) if so desire.

Specifically, the training of the BPN is an iteration of the following two-phase process:

• The feedforward pass:

A randomly selected sample x_{n} labeled by y_{n} is presented to the input layer and forwarded through the weighted connections to the hidden layer and then the output layer to produce output ŷ = f(x, W^{h}, W^{o}).

• The backward error backpropagation:

The squared error ε = ||y_{n} − ŷ_{n}||^{2}/2 measuring the difference between the actual output ŷ_{n} and the desired output y_{n} as the target is propagated backward from the output layer through the hidden layer to the input layer, during which the weights of both the output and hidden layers are modified so that ε will be reduced when the same or similar pattern is presented to the input layer in the future.

Figure 18.3 Back Propagation Network

This two-phase process is iteratively carried out until eventually the error is minimized for all training samples and BPN is properly trained.

We now consider the specific computation taking place in both the forward and backward passes.

• The forward pass from input x = [x_{1}, ⋯, x_{d}]^{T} represented to the input layer to the output ŷ_{n} = [y_{1}, ⋯, y_{m}]^{T} produced by the output layer:

– From input layer to hidden layer:

z_{j} = g(a_{j}^{h}) = g (∑_{k=1}^{d} w_{jk}^{h}x_{k} + b_{j}^{h}) = g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) = g(x^{T} w_{j}^{h}) (j = 1, ⋯, l)

(18.17)

where both x = [x_{0} = 1, x_{1}, ⋯, x_{d}]^{T} and w_{j}^{h} = [w_{j0}^{h} = b_{j}^{h}, w_{j1}^{h}, ⋯, w_{jd}^{h}]^{T} are augmented, and a_{j}^{h} = x^{T} w_{j}^{h} is the activation of the jth hidden layer node. These l equations can be expressed in vector form:

z = g (W^{h}x)  (18.18)

where z = [z_{1}, ⋯, z_{l}]^{T}, and W^{h} = [w_{1}^{h}, ⋯, w_{l}^{h}]^{T} is an l × (d + 1) matrix.

– From hidden layer to output layer:

ŷ_{i} = g(a_{i}^{o}) = g (∑_{j=1}^{l} w_{ij}^{o}z_{j} + b_{i}^{o}) = g (∑_{j=0}^{l} w_{ij}^{o}z_{j})

= g (∑_{j=0}^{l} w_{ij}^{o}g (∑_{k=0}^{d} w_{jk}^{h}x_{k})) (i = 1, ⋯, m) (18.19)

which can also be written in vector form as:

ŷ_{i} = g(z^{T} w_{i}^{o})  (18.20)

where both z = [z_{0} = 1, z_{1}, ⋯, z_{l}]^{T} and w_{i}^{o} = [w_{i0}^{o} = b_{i}^{o}, w_{i1}^{o}, ⋯, w_{il}^{o}]^{T} are augmented, and a_{i}^{o} = z^{T} w_{i}^{o} is the activation of the ith output layer node. These m equations can be expressed in vector form:

ŷ = g (W^{o}z)  (18.21)

where ŷ = [ŷ_{1}, ⋯, y_{m}]^{T}, and W^{o} = [w_{1}^{o}, ⋯, w_{m}^{o}]^{T} is an m × (l + 1) matrix.

• The backward error propagation from error ε to the input layer:

The squared error ε = ||y − ŷ||^{2}/2 can be written as a function of the output layer weights w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) and hidden layer weights w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d):

ε = \frac{1}{2}||y − ŷ||^{2} = \frac{1}{2} ∑_{i=1}^{m}(ŷ_{i} − y_{i})^{2} = \frac{1}{2} ∑_{i=1}^{m}[g(a_{i}^{o}) − y_{i}]^{2}

= \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o}z_{j}) − y_{i}]^{2} = \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o} g(a_{j}^{h})) − y_{i}]^{2}

= \frac{1}{2} ∑_{i=1}^{m} [g (∑_{j=0}^{l} w_{ij}^{o} g (∑_{k=0}^{d} w_{jk}^{h}x_{k})) − y_{i}]^{2}  (18.22)

Similar to the objective function of the ridge regression considered in Section 4.2, an additional regularization term weighted by the weight decay parameter λ can be included in the objective function to penalize large weights and thereby prevent overfitting:

J(W^{h}, W^{o}) = ε + \frac{λ}{2} [||W^{h}||_{2}^{2} + ||W^{o}||_{2}^{2}]  (18.23)

where the two weight matrices in the regularization term should exclude the first column for the bias term b which need not to be forced to be small, and ||W||_{2} = \sqrt{∑_{i} ∑_{j} w_{ij}^{2}} is the Frobenius norm (Section A.4.2).

The gradient descent method is used to modify first the output layer weights and then the hidden layer weights to iteratively reduce the objective function J in the following steps:

– Find the gradient of J with respect to the output layer weights w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) by the chain rule:

\frac{∂ J}{∂ w_{ij}^{o}} = \frac{∂ ε}{∂ ŷ_{i}} \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ w_{ij}^{o}} + λw_{ij}^{o} = (ŷ_{i} − y_{i}) g′(a_{i}^{o}) z_{j} + λw_{ij}^{o}
= −δ_{i} g′(a_{i}^{o}) z_{j} + λw_{ij}^{o}  (18.24)

where δ_{i} = y_{i} − ŷ_{i}.

– Update w_{ij}^{o} (i = 1, ⋯, m, j = 0, 1, ⋯, l) to reduce J by gradient descent method with learning rate or step size η:

w_{ij}^{o} ⇐ w_{ij}^{o} − η \frac{∂J}{∂w_{ij}^{o}} = w_{ij}^{o} − η (−δ_{i} g′(a_{i}^{o}) z_{j} + λw_{ij}^{o})
= w_{ij}^{o} − η (−d_{i}^{o} z_{j} + λw_{ij}^{o})  (18.25)

where d_{i}^{o} = δ_{i}g′(a_{i}^{o}), or in matrix form:

W^{o} ⇐ W^{o} − η (−d^{o}z^{T} + λW^{o}) = W^{o} + ηd^{o}z^{T} − ηλW^{o} (18.26)

where d^{o} = [d_{1}^{o}, ⋯, d_{m}^{o}]^{T} is the elementwise (Hadamard) product δ ⊙ g′(a^{o}), and d^{o}z^{T} = d^{o} ⊗ z is the outer product of d^{o} and z = [z_{0} = 1, z_{1}, ⋯, z_{l}].

– Find the gradient of J with respect to the hidden layer weights w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d) by the chain rule:

\frac{∂ J}{∂ w_{jk}^{h}} = \frac{∂ ε}{∂ ŷ_{i}} \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ z_{j}} \frac{∂ z_{j}}{∂ a_{j}^{h}} \frac{∂ a_{j}^{h}}{∂ w_{jk}^{h}} + λw_{jk}^{h} = ∑_{i=1}^{m} (ŷ_{i} − y_{i}) \frac{∂ ŷ_{i}}{∂ a_{i}^{o}} \frac{∂ a_{i}^{o}}{∂ z_{j}} \frac{∂ z_{j}}{∂ a_{j}^{h}} \frac{∂ a_{j}^{h}}{∂ w_{jk}^{h}} + λw_{jk}^{h}

= − ∑_{i=1}^{m} δ_{i}g′(a_{i}^{o})w_{ij}^{o} g′(a_{j}^{h})x_{k} + λw_{jk}^{h} = − (∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) g′(a_{j}^{h})x_{k} + λw_{jk}^{h}
= −δ_{j}^{h} g′(a_{j}^{h})x_{k} + λw_{jk}^{h} = −d_{j}^{h}x_{k} + λw_{jk}^{h}  (18.27)

where we have defined

δ_{j}^{h} = ∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}, d_{j}^{h} = δ_{j}^{h}g′(a_{j}^{h}) = (∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) g′(a_{j}^{h}), (j = 1, ⋯, l)

(18.28)

or in matrix form:

[ δ_{1}^{h} ; ⋮ ; δ_{l}^{h} ] = [ \begin{matrix}w_{11}^{o} & ⋯ & w_{1m}^{o} \\ ⋮ & ... & ⋮ \\ w_{l1}^{o} & ⋯ & w_{lm}^{o} \end{matrix} ] [ d_{1} ; ⋮ ; d_{m} ] = W^{o}d^{o}  (18.29)

Here W^{o} is an m × l matrix, the same as that defined above but with the first column of b_{j}’s removed.

– Update w_{jk}^{h} (j = 1, ⋯, l, k = 0, 1, ⋯, d) to reduce J by gradient descent method:

w_{jk}^{h} ⇐ w_{jk}^{h} − η \frac{∂J}{∂w_{jk}^{h}} = w_{jk}^{h} − η (−δ_{j}^{h} g′(a_{j}^{h})x_{k} + λw_{jk}^{h})
= w_{jk}^{h} − η (−d_{j}^{h}x_{k} + λw_{jk}^{h})  (18.30)

or in matrix form:

W^{h} ⇐ W^{h}−η (−W^{o}d^{o} ⊙ g′(a^{h})x^{T} + λW^{h}) = W^{h}+ηd^{h}x^{T} −ηλW^{h}

(18.31)

where d^{h} = W^{o}d^{o} ⊙ g′(a^{h}) is the elementwise product of vector W^{o}d^{o} and g′(a^{h}), and d^{h}x^{T} = d^{h} ⊗ x is the outer product of vectors

d^{h} and x.

In summary, here are the steps in each iteration:

1. Input a randomly selected pattern [x_{1}, ⋯, x_{d}]^{T}, construct d + 1 dimensional vector x = [1, x_{1}, ⋯, x_{d}]^{T};

2. Compute z = g(W^{h}x), and construct l + 1 dimensional vector z ← [1, z];

3. Compute ŷ = g(W^{o}z);

4. Get elementwise product d^{o} = (y − ŷ) ⊙ g′(a^{o}) = δ ⊙ g′(a^{o});

5. Update output weights W^{o} ⇐ W^{o} + η d^{o}z^{T} − ηλW^{o};

6. Get elementwise product d^{h} = W_{m×l}^{o}d^{o} ⊙g′(a^{h}) = δ^{h} ⊙g′(a^{h}), where W_{m×l}^{o} is the same as W^{o} but with its first column removed.

7. Update hidden weights W^{h} ⇐ W^{h} + η d^{h}x^{T} − ηλW^{h};

8. Terminate the iteration if the error ε is acceptably small for all of the training patterns. Otherwise repeat the above with another pattern in the training set.

The Matlab code for the essential part of the BPN algorithm is listed below. Array X contains C classes each with K samples, array Y are the labelings of the C ∗ K training samples, array W contains the N + 1 dimensional weight vectors for the M output nodes. Also, L is the number of hidden nodes, eta is the learning rate between 0 and 1, and tol is the tolerance for the termination of the learning iteration (e.g., 0.01).

function [Wh, Wo, g]=backPropagate(X,Y)
    syms x;
    g=1/(1+exp(-x));             % Sigmoid activation function
    dg=diff(g);                  % its derivative function
    g=matlabFunction(g);
    dg=matlabFunction(dg);
    [d,N]=size(X);               % numbers dimensions and samples
    X=[ones(1,N); X];            % augment X by adding a row of x0=1
    Wh=1-2*rand(L,d+1);          % Initialize hidden layer weights
    Wo=1-2*rand(M,L+1);          % Initialize output layer weights
    er=inf;
    while er > tol
        I=randperm(N);           % random permutation of samples
        er=0;
        for n=1:N                % for all N samples for an epoch
            x=X(:,I(n));         % pick a training sample
            ah=Wh*x;             % activation of hidden layer
            z=[1; g(ah)];        % augment z by adding z0=1
            ao=Wo*z;             % activation to output layer
            yhat=g(ao);          % output of output layer
            delta=Y(:,I(n))-yhat % delta error

            er=er+norm(delta)/N  % test error
            do=delta.*dg(ao);
            Wo=Wo+eta*do*z';     % update weights for output layer
            dh=(Wo(:,2:L+1)'*do).*dg(ah);   % delta of hidden layer
            Wh=Wh+eta*dh*x';     % update weights for hidden layer
        end
    end
end

The training process of BP network can also be considered as a data modeling problem as in regression to fit the given data {(x_{i}, y_{i}), (i = 1, ⋯, N)} by a function with the weights of both the hidden and output layers as the parameters:

y = f(x, W^{h}, W^{o})  (18.32)

The goal is to find the optimal parameters W^{h} and W^{o} that minimize the difference r = y −ŷ between the desired and the actual outputs. The Levenberg-Marquardt algorithm discussed previously can be used to obtain the parameters

(e.g., the Matlab function trainlm http://www.mathworks.com/help/nnet/ref/trainlm.html).

The three-layer BPN containing a single hidden layer discussed above can be easily generalized to a multilayer BPN containing any number of hidden layers (e.g., for a deep learning network) by simply repeating steps 6 and 7 for the single hidden layer in the algorithm list above. We assume in addition to the input layer, there are in total L learning layers including all the hidden layers and the output layer, indexed by l = 1, ⋯, L. Then step 6 above becomes:

d^{(l)} = δ^{(l)} · g′(a^{(l)}) (l = 1, ⋯, L)  (18.33)

where a^{(l)} = (W^{(l)}z^{(l)}) is the activation of all nodes in the lth layer, z^{(l)} = W^{(l−1)}z^{(l−1)} is the input to the lth layer, output from the (l-1)th layer (z^{(1)} = x), and

δ^{(l)} = \{ \begin{matrix}y − ŷ & (l = L) \\ W^{(l)}d^{(l−1)} & (1 ≤ l < L) \end{matrix}  (18.34)

Then the same step 7 for updating the weights can be carried out based on the gradient vector:

d^{(l)} ⊗ z^{(l)} = d^{(l)}(z^{(l)})^{T}  (18.35)

The Matlab code segment below is for the BP network of multiple hidden layers, in which L is the total number of learning layers and m(l) is the number of nodes in the lth layer (l = 1, ⋯, L):

W={1-2*rand(m(1),d+1)};           % initial weights for first layer
for l=2:L
    W{l}=1-2*rand(m(l),m(l-1)+1); % initial weights for other layers
end
er=inf;
d={};

while er > tol
    I=randperm(N);                % random permutation of samples
    er=0;
    for n=1:N                     % N samples for an epoch
        z={1;X(:,I(n))};          % pick a training sample
        a={W{1}*z{1}};            % activation of first layer
        for l=2:L                 % the forward pass
            z{l}=[1;g(a{l-1})];   % input to the lth layer
            a{l}=W{l}*z{l};       % activation of the lth layer
        end
        yhat=g(a{L});             % actual output of last layer
        delta=Y(:,I(n))-yhat;     % delta error
        er=er+norm(delta)/N;      % test error
        d{L}=delta.*dg(a{L});     % d for output layer
        W{L}=W{L}+eta*d{L}*z{L}'; % upddate weights for output layer
        for l=L-1:-1:1            % the backward pass
            d{l}=(W{l+1}(:,2:end)'*d{l+1}).*dg(a{l}); % d of hidden layers
            W{l}=W{l}+eta*d{l}*z{l}'; % upddate weights for hidden layers
        end
    end
end

Example 18.4 Figs. 18.4 and 18.5 show the classification results of two 2-D datasets with 2 and 3 classes previously used in Examples 13.6 and 13.7 for the naive Bayes classifier (with quadratic decision boundaries). The error rates are respectively 13% and 11.5%, and the confusion matrices are shown below. Iinterestingly, in the case of the 3-class dataset, there is a green region sandwiched between the blue and red ones in the results generated by both the naive Bayes and back propagation methods based on very different learning algorithms.

[ \begin{matrix}176 & 24 \\ 22 & 178 \end{matrix} ]  [ \begin{matrix}185 & 14 & 1 \\ 5 & 181 & 14 \\ 11 & 33 & 156 \end{matrix} ]

Example 18.5 The back propagation network is applied to the classification of the handwritten digit dataset. Half of the N = 2240 samples in the dataset is used for training while the other half used for testing. The classification results are shown by the two confusion matrices below, when the network is tested on half of the samples used for training (left) and on the remaining half reserved for testing (right). Note that the error rate 74/1120 = 6.6% in the latter case is understandably higher than the error 27/1120 = 2.4% for the former, as the network may overfit half of the samples used for training and does not generate too well to the other half of the samples it has never seen.

18.2 Back Propagation 437

Figure 18.4 BP Network Classification of the XOR Dataset

[ \begin{matrix}113 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\ 0 & 107 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 1 & 106 & 1 & 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 4 & 113 & 0 & 0 & 0 & 0 & 0 & 3 \\ 0 & 0 & 0 & 0 & 113 & 8 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 1 & 120 & 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 102 & 0 & 0 & 0 \\ 0 & 1 & 2 & 0 & 0 & 1 & 0 & 103 & 0 & 1 \\ 0 & 1 & 0 & 1 & 2 & 0 & 0 & 0 & 104 & 1 \\ 1 & 1 & 0 & 0 & 2 & 0 & 0 & 0 & 0 & 112 \end{matrix} ]
[ \begin{matrix}100 & 1 & 0 & 0 & 0 & 4 & 3 & 0 & 0 & 0 \\ 0 & 111 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 1 \\ 1 & 0 & 113 & 1 & 0 & 0 & 0 & 1 & 0 & 0 \\ 2 & 1 & 1 & 100 & 0 & 0 & 0 & 1 & 0 & 3 \\ 0 & 1 & 0 & 0 & 105 & 0 & 0 & 0 & 2 & 3 \\ 1 & 0 & 0 & 0 & 3 & 95 & 0 & 0 & 2 & 0 \\ 0 & 2 & 1 & 0 & 0 & 0 & 118 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1 & 1 & 0 & 0 & 105 & 1 & 5 \\ 1 & 1 & 2 & 1 & 1 & 5 & 0 & 2 & 97 & 5 \\ 0 & 2 & 0 & 0 & 4 & 0 & 0 & 0 & 0 & 102 \end{matrix} ]

Before leaving this section, we further make the following comments regard-

Figure 18.5 BP Network Classification of a Three-Class Dataset

ing how the back propagation network, and in general, multilayer feed forward network, is closely related to the general method of regression analysis discussed before.

• The computational model for a single neuron in Eq. (17.3) based on the logistic activation function in Eq. (17.5) is actually the same as Eq. (7.6) in logistic regression in Section 7.1:

g (∑_{j=1}^{d} w_{j}x_{j} + b) = g(w^{T} x) = \frac{1}{1 + e^{−w^{T} x}}  (18.36)

treated as the conditional probability p(ŷ = 1|x, w). Both methods are to determine the model parameter w, but based on different methods.

• A supervised learning neural network can be considered as a function ŷ = f(x, w) as a model of the training dataset D = {X, y}, while the parameter w can be found by solving an optimization problem (Chapter 5), by either the least squares method that minimizes ε = ||y − ŷ||^{2}, or the maximum likelihood method that maximizes L(w) = P (y|X, w), equivalent to minimizing −l(w) = − log L(w), which can be interpreted as the cross entropy between the model distribution based on D and the unknown ground true distribution, as discussed in Section 4.1.

• As a neural network can be considered as a model for the function y = f(x) relating y to x in the given dataset D, a fundamental question to ask is whether such a network is able to model any function. This question is answered by the universal approximation theorem, stating that a feedforward network with arbitrary width (number of nodes in a hidden layer) or depth (number of hidden layers) can represent any function with arbitrary accuracy.

While this theorem is formally proven, it can also be understood intuitively. Recall the method of linear regression based on basis functions in Chapter 5, by which a function is approximated by a repression function as a linear combination of K basis functions in Eq. (5.47):

ŷ_{m} = ∑_{k=1}^{K} w_{mk}^{o} φ_{k}(x) = ∑_{k=1}^{K} w_{mk}^{o} g (∑_{i=0}^{d} w_{ki}^{h}x_{i})  (18.37)

which is in the same form as in Eq. (18.19) for the three-layer feed forward network discussed above, with a linear activation function for the output layer. We see the the output of the hidden layer nodes are the basis functions, and the output of an output layer node is a linear combination of such basis functions. As shown in Example 5.10 there, a given function can be approximated with progressively higher accuracy as more basis functions, i.e., more hidden layer nodes, are used, if the width of the network is unbounded.

If the width of the network is bounded but its depth is not, we can alternatively increase the number of layers of the network so that in the equation above we can have as many levels of summations as needed, i.e., the total number of terms in these summations can increased without a bound, then by the same argument, such a network can again approximate a given function with arbitrary accuracy. But as a word of warning, while trying to match the given dataset for better accuracy, watch out for the overfitting problem.

18.3 Autoencoder

Autoencoder is a neural network method that learns to represent the patterns in a given dataset in such a way that both the dimensionality and the noise can be reduced, and it can therefore be considered as a method for feature selection and dimensionality reduction, similar to the KLT method.

An autoencoder as shown in Fig. 18.6 is similar to a back propagation network in that it iteratively updates the weights in all hidden and output layers by back propagation so that eventually its outputs are as desired. But different from a back propagation network that learns to associate a pattern x_{n} to another pattern y_{n} as its labeling, the autoencoder is an unsupervised learning method based on unlabeled data, and it learns to associate each pattern x_{n} to itself.

The purpose of the autoencoder is not to simply copy the data. Instead, we

Figure 18.6 Thee layer Network for Autoenoder

desire to reduce the number of hidden layer nodes as much as possible, by imposing constraints such as sparsity to the hidden layer(s) in terms of the number of active nodes, so that there are many fewer nodes in the hidden layers when compared to the input and output layers. By doing so, the hidden layers becomes the bottleneck of the multi-layer network, which can then be considered as an encoder by which the dimensionality of the data presented to the input layer to be reproduced by the output layer is converted into a code in a much reduced dimensional space in the hidden layer. By this approach, the data can be much compressed while the information contained in it is mostly preserved, and certain potential structures inherent in the dataset may be revealed, very much like dimensionality reduction achieved PCA based on KLT.

The computation taking place in an autoencoder can therefore be considered as an encoding/decoding process, with encoding taking place between the input and hidden layers, and decoding taking place between the hidden and output layers. Due to the reduced number of nodes in the hidden layers, the data size can be significantly reduced for data compression, if some of the components of data samples x in the dataset are correlated, i.e., there is some redundancy in the data.

While the methods of autoencoder and PCA based on KLT are similar to each other in that both can compress the data without losing much information, they are different in that the PCA is a linear method, as the principal components are linear combinations of the data components, while the signals represented by the hidden layer of the autoencoder are nonlinear functions of the data components (due to the nonlinear activation function). Consequently the autoencoder can be a more powerful method than the PCA, but with the cost of higher computational complexity.

We now consider specifically how an autoencoder works. Same as in back propagation, a data vector x presented to the input layer is forward propagated first to the hidden layer (same as Eq. (18.18)):

z = g(W^{h}x),  or z_{j} = g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) (j = 1, ⋯, l)  (18.38)

and then further to the output layer (same as Eq. (18.21)):

x̂ = g(W^{o}z)  or x̂_{i} = g (∑_{j=0}^{l} w_{ij}^{o}z_{k}) (i = 1, ⋯, m)  (18.39)

Also same as in back propagation, the weights in both W^{h} and W^{o} are updated during training in the back propagation process so that the following objective function for the squared error between the input and desired output, the same as the input, is minimized:

ε = \frac{1}{2}||x − x̂||^{2}  (18.40)

Here we assume there is only a single hidden layer for convenience, but the computation for both the forward and backward passes can be easily extended to multiple layers.

By limiting the number l of the hidden layer to be smaller than that of input and output layer, the dimensionality of the data can be reduced from d for the dimensionality of the input x = [x_{1}, ⋯, x_{d}]^{T} to l, the number of hidden layer nodes, while most of the information in the original data is reserved.

It is interesting to compare the autoencoder and the PCA based on KLT transform in terms of how they map a given data vector x in the original d-dimensional space to a lower-dimensional space. Recall the KLT previously considered:

y = [ y_{1} ; ⋮ ; y_{d} ] = V^{T} x = [  v_{1}^{T} ; ⋮ ; v_{d}^{T}  ] x = [ v_{1}^{T} x ; ⋮ ; v_{d}^{T} x ]  (18.41)

In KLT, the ith component y_{i} = v_{i}^{T} x of y is the inner product of the ith eigenvector v_{i} and the input x, the projection of x onto v_{i} if it is normalized. On the other hand, in a neural network, the inner product of the weight vector w_{i} and the input x is further mapped to z_{i} = g(w_{i}^{T} x). If, specially, the activation function g(u) = u is linear (an identical function), the weight vectors w_{i} play the same role as the eigenvectors v_{i} as the basis vectors that span the space, i.e., autoencoder and PCA are essentially the same. But as in general, the activation function g(u) is nonlinear, the autoencoder is a nonlinear mapping, of which the linear KLT (a rotation in the space) is a special case.

Dimension reduction exemplified by the KLT is effective if the components of the pattern vectors are highly correlated, i.e., some of them may be redundant as they carry similar information. However, if all components carry their own independent information, dimensionality reduction will inevitably cause significant information loss. Similarly in an autoencoder, if the components of the data points are mostly independent of each other with little redundancy, they may not be reproduced as the output of the autoencoder if the number of hidden layer nodes is too small.

When the data components are correlated, it is possible to achieve the desired low sparsity of the hidden layer of the autoencoder by using only a small number of hidden nodes, or allowing only a small fraction of them to be active (over different input patterns). The second approach can be realized by (a) keeping only a small number of hidden nodes with maximum outputs while setting the rest to zero, or (b) forcing the average activation of the hidden nodes to be low. This can be achieved by modify the regularization term in Eq. (18.23) for small weights so that it acts to force the average activation level low, as either of the following:

• Minimize the following L1 or L2 average of all activations:

J = ε + λ ∑_{j=1}^{l} |a_{j}|, J = ε + \frac{λ}{2} ∑_{j=1}^{l} a_{j}^{2}  (18.42)

• Minimize the difference between the actual and desired average activation:

J = ε + ∑_{j=1}^{l} KL(ρ||ρ̂_{j})

= ε + ∑_{j=1}^{l} [ρ log (\frac{ρ}{ρ̂_{j}} ) + (1 − ρ) log (\frac{1 − ρ}{1 − ρ̂_{j}} )]  (18.43)

Here ρ is the desired average activation of the hidden layer, a small value close to zero (e.g., ρ = 0.05), and ρ̂_{j} is the actual average activation of the jth hidden node over all N samples in the given dataset:

ρ̂_{j} = \frac{1}{N} ∑_{n=1}^{N} ρ_{j}(x_{n}) = \frac{1}{N} ∑_{n=1}^{N} g(a_{j}) = \frac{1}{N} ∑_{n=1}^{N} g (∑_{k=0}^{d} w_{jk}^{h}x_{k}) (j = 1, ⋯, l)

(18.44)

where ρ_{j}(x_{n}) = g(a_{j}) = g (∑_{k} w_{jk}^{h}x_{k}) is the activation level of the node while responding to the nth input sample x_{n}. Also, ρ is treated as the probability for some binary random variable (with Bernoulli distribution) to be 1, and 1−ρ the probability for it to be 0, and so is ρ̂_{j} for another binary variable. We therefore see that the regularization term in Eq. (18.43) is the sum of the KL-divergence (Section B.2.2) between the two distributions for all hidden nodes, measuring the overall difference between the desired and actual sparsities in terms of ρ and ρ̂. By minimizing the objective function including such a regularization term, the sparsity of the hidden layer in terms of the average activation is forced to be as low as desired.

The plot in Fig. 18.7 compares the KL-divergence and the quadratic

Figure 18.7 Comparison of KL-divergence and Quadratic Function

function a(ρ − ρ̂)^{2}. While both functions are minimized to zero when ρ̂ = ρ = 0.3, they behave differently in the range of [0, 1] in that the KLdivergence becomes significantly larger than the quadratic function when ρ̂ are very different from ρ = 0.3.

Now the objective function for the autoencoder can be written as the sum of the error ε and the regularization term above:

J(W^{h}, W^{o}) = ε + λ ∑_{j=1}^{l} KL(ρ||ρ̂_{j})  (18.45)

During the iterative learning the weights are updated based on gradient descent method, same as in back propagation. Given a specific input x we first get the output x̂ through the forward pass, and then modify the weights of both the output and hidden layers in the backward pass based on the gradient of the objective function ε = ||x − x̂||^{2}/2 with respect to the weights. While the computation for updating the weights for the output layer is the same as steps 4 and 5 in the algorithm of back propagation listed in Section 18.2, the computation for the hidden layer weights as shown below is different from steps 6 and 7 for back propagation due to the regularization term:

\frac{∂}{∂w_{jk}^{h}} J(W^{h}, W^{o}) = \frac{∂ε}{∂w_{jk}^{h}} + λ\frac{∂}{∂w_{jk}^{h}} [∑_{j=1}^{l} KL(ρ||ρ̂_{j})]  (18.46)

The first term above is the same as as that in Eq. (18.27), while the second term for regularization based on the KL-divergence is different:

\frac{∂}{∂ρ̂_{j}} [∑_{j=1}^{l} ρ log (\frac{ρ}{ρ̂_{j}} ) + (1 − ρ) log (\frac{1 − ρ}{1 − ρ̂_{j}} )] \frac{∂ρ̂_{j}}{∂w_{jk}^{h}}

= [ρ\frac{∂}{∂ρ̂_{j}} (log ρ − log ρ̂_{j}) + (1 − ρ)\frac{∂}{∂ρ̂_{j}} [(log(1 − ρ) − log(1 − ρ̂_{j}))]] \frac{∂}{∂w_{ij}} g (∑_{k=0}^{d} w_{jk}^{h}z_{i})
= (−\frac{ρ}{ρ̂_{j}} + \frac{1 − ρ}{1 − ρ̂_{j}} ) g′(a_{j})x_{k}  (18.47)

Now d_{j}^{h} in Eq. (18.27) for back propagation (defined in Eq. (18.28) ) can be modified as below for the autoencoder:

d_{j}^{h} = [(∑_{i=1}^{m} d_{i}^{o}w_{ij}^{o}) + λ (−\frac{ρ}{ρ̂_{j}} + \frac{1 − ρ}{1 − ρ̂_{j}} )] g′(a_{j}^{h})  (18.48)

Using this modified d_{j}^{h} in step 6 of the algorithm for the propagation listed in the previous section, we get the algorithms for the autoencoder.

The Matlab code segment below is the essential part of the autoencoder algorithm, which takes the given dataset as input and generates the weights for both hidden and output layers as output. Here to achieve the desired sparsity of the hidden layer, the first approach mentioned above is used, i.e., out of all L hidden layer nodes, La < L most active ones are kept while all others are set to zero. Alternatively, the more elaborate approach based on KL-divergence can be used as well.

Wh=1-2*rand(L,d+1);                % hidden layer weights
Wo=1-2*rand(d,L+1);                % output layer weights
done=0;
it=0;
while ~done
    it=it+1;
    X=Xm;
    [d N]=size(X);
    I=randperm(N);                 % random order of N samples
    Ah=0;
    for n=1:N                      % for each samples
        x=X(:,I(n));               % input vector
        x1=[1;x];                  % augmented input vector
        ah=Wh*x1;                  % activation of hidden nodes
        z=g(ah);                   % output of hidden layer
        zsorted=sort(z,'descend'); % sort hidden layer output
        z=z.*(z>=zsorted(La));     % keep La most active nodes
        z=[1;z];                   % augment hidden layer output
        ao=Wo*z;                   % activation of output nodes

        xhat=g(ao);                % actual output of output layer
        do=(x-xhat).*dg(ao);       % delta of output layer
        Wo=Wo+eta*do*z';           % update output layer weights
        dh=(Wo(:,2:L+1)'*do).*dg(ah);  % delta of hidden layer
%        dh=((Wo(:,2:L+1)'*do)+lambda*(-r./rp+(1-r)./(1-rp))).*dg(ah);
        Wh=Wh+eta*dh*x1';          % update hidden layer weights
    end
end

Recall that in dimensionality reduction based on KLT, the transform matrix is the eigenvector matrix of either the covariance matrix Σ_{x} of the entire dataset X = [x_{1}, ⋯, x_{N}], or the between-class scatter matrix S_{B} in the case of supervised classification. Similarly an autoencoder can also be trained based on either the entire dataset X for dimensionality reduction, or a dataset X = [m_{1}, ⋯, m_{K}] containing the mean vectors of some K classes (calculated based on labeled training set) for classification.

In the following examples, we compare the performances of both the autoencoder (trained on a dataset composed of only mean vectors of all classes) and the method PCA based on the between-class scatter matrix of the classes, when they are applied to reduce the dimensionality of the given dataset to be classified by the naive Bayes classifier.

Example 18.6 Both methods of autoencoder and PCA are applied to a dataset containing K = 8 classes each in one of the octants of the 3-D space, and then compared in terms of the separability and classification error rate in the resulting lower dimensional space. Note that in this case all three dimensions are equally important for representing the 8 classes, as also reflected by the eigenvalues 0.37%, 0.33%, 0.30% of the covariance matrix, indicating that roughly the same amount of information is contained in each of the three components of the data. We see that it is difficult to reduce the dimensionality without losing significant amount of information.

Fig. 18.8 shows the dataset after its dimensionality if reduced from 3 to 2 by autoencoder (left) and PCA (right) methods. by visual observation, We see that the data points are similarly distributed in the resulting 2-D space, but the class separability for the autoencoder seems to be slightly better than that by the PCA. This result can be confirmed quantitatively as shown in the first row of the Table in Eq. (18.49), where the separability J_{B/T} = tr (S_{T}^{−1}S_{B}) (Eq. (9.24)) and the error rates for classification (by naive Bayes classifier) are listed to left and right of the slash symbol for the autoencoder and PCA methods, respectively. showing the autoencoder has a slightly high separability and lower error rate then the PCA method.

Figure 18.8 Autoencoder (left) vs. PCA (right) for Dimension Reduction for a 3-D Dataset

Figure 18.9 Autoencoder (left) vs. PCA (right) for Dimensionality Reduction for Iris dataset

Separability

Error rate

8-class Dataset 1.52/1.51 0.304/0.320

Iris Dataset  1.12/1.07  0.020/0.027

Example 18.7 The autoencoder and PCA methods are also compared when applied to the iris dataset containing three classes in the 4-D space. The resulting 2-D dataset is shown in Fig. 18.9, and the separabilities the classification error rates of the two methods are compared in the second row of Table 18.49. We see again that the autoencoder has a little higher separability and lower error rate than the PCA method.

Based on the separabilities and classification error rates listed in Table 18.49, we see that for both datasets, the performance of the autoencoder (to the

Figure 18.10 Autoencoder (Left) vs. KLT (right) for Dimension Reduction

Example 18.8 The autoencoder network is typically trained by all samples in the dataset. However, if these samples in a dataset are labeled to belong to one of a set of classes, the autoencoder can also be trained by these classes. Specifically in this example, the three classes in the iris dataset each represented by the average of all samples in the class are used to train the autoencoder. Fig. 18.10 shows the 4-D iris dataset mapped to 2-D (top) and 3-D (bottom) spaces by both the autoencoder (left) and KLT (right). It is interesting to observe that in the case of autoencoder (left), the data points representing the three classes are arranged into a line structure in both 2 and 3-D spaces.

Example 18.9 The same method is applied to the dataset of 10 handwritten digits from 0 to 9, each represented by 224 training samples in image form of 16 × 16 pixels, treated as a 256 dimensional vector. After the autoencoder network is fully trained based on the means of all 224 samples of each of the 10 classes, the dataset containing 224×10 = 2240 data samples in the l-dimensional space spanned by the hidden layer nodes that encode the 256-d samples is then visualized in 3-D space based the KLT transform, as shown in Fig. 18.11, where the dataset in the 256-D space is mapped to l = 3, 4, 5, 6 dimensional space by the autoencoder (with l hidden layer nodes) as well as the KLT method, and then further mapped down to 3-D by KLT for visualization.

Also, as shown in Fig. (18.12) for the plots of the separabilities of the ten

Figure 18.11 Dimensionality Reduction by Autoencoder and KLT

Figure 18.12 Separability in Lower Dimensional Space

classes in the lower dimensional space, the autoencoder and KLT method achieve comparable separabilities.

18.4 Deep Learning

Deep learning, a method based on multi-layer feed forward network with multiple hidden layers, has gained popularity and caused a new wave of enthusiasm for artificial neural networks in the recent years, due heavily to the progress in computational capability as well as the latest development in new algorithms.

Conceptually, deep learning network can be simply understood as a network composed of multiple learning layers, including more than one hidden layer as well as the output layer. The perceptron network with a single learning layer and the three-layer network with the hidden and output learning layers discussed previously can be considered as special cases of deep learning. If we model a single learning layer by a nonlinear function that maps an input x to an output

y = g(w^{T} x) = f(x, w)  (18.49)

than an L-layer deep learning network can simply be described by

y = f^{(L)}(f^{(L−1)}(⋯ f^{(1)}(x, w^{(1)}), w^{(L−1)}), w^{(L)}) = f(x, W)  (18.50)

where f represents the cascade of all the L single-layer mapping functions f^{(1)} to f^{(L)}, and W = [w^{(1)}, ⋯, w^{(L)}] is a matrix containing all weights in the network. When properly trained, such a multi-layer learning network will be able to model some highly complicated and nonlinear relationship between input x and output y.

When the multiple layers in a deep learning network are fully connected, i.e., a node in a layer gets inputs from all nodes in the previous layer weighted by w, then the method of back propagation discussed previously can be used to update the weights based on gradient descent.

The basic structure of the multi-layer network has been inspired by the biology of the brain. The hierarchical structure of the network is composed of lower level layers where the neurons selectively respond to some simple local features in the data, while neurons in subsequent higher level layers selectively respond to progressively more complex and global patterns. Such a hierarchy is very similar to the two main visual pathways in the primate brain, known as the dorsal or “where” pathway and the ventral or “what” pathway, both formed by multiple levels of cortical areas starting from the primary visual cortex (V1), each composed of different types of neurons in terms of the size and specific type of the visual stimuli they are selectively responsive to.

Specifically, the hierarchy of ventral pathway known for visual object recognition is composed of neurons in the lower layer such as V1 where the simple cells with small receptive field selectively respond to edges of different orientations, and neurons with progressively larger receptive fields in the higher layers selectively respond to more complex features in larger areas of the visual field, such as neurons in the inferior temporal (IT) area sensitive to visual patterns such as image of faces. This visual pathway is shown in Figs. 18.13 (“Understanding the Pathophysiology of Alzheimer’s Disease and Mild Cognitive Impairment: A Mini Review on fMRI and ERP Studies” Neurology Research International. 2012, Article ID 719056) and 18.14 (“Feature integration and object representations along the dorsal stream visual hierarchy”,Frontiers in Computational Neuroscience, 05 August 2014, Vol 8 - 2014).

As an example of deep learning networks, the convolutional neural network (CNN) is hierarchical structure composed of a sequence of multiple 2-D layers for the purpose of image object recognition. This network can be considered to a large extent a model that mimics the ventral pathway of the visual cortex described above, in the sense that the nodes in the lower layers detect and extract local features in the visual data (mimicking the simple cells in V1), while the nodes in higher layers respond selectively to more global patterns represented in

Figure 18.13 Visual Pathways

Figure 18.14 Specialization of Visual Cortical Areas

terms of the local features detected by the lower layers (mimicking the cells in IT).

Specifically, in the lower layers of the CNN, local image features such as edges are extracted by convolution, a basic technique commonly used in image processing. Mathematically, a 1-D continuous convolution in time domain is an operation defined for two functions f(t) and g(t):

y(t) = x(t) ∗ h(t) = ∫_{−∞}^{∞} x(t − τ)h(τ)dτ = ∫_{−∞}^{∞} h(t − τ)x(τ)dτ  (18.51)

or a 2-D discrete convolution is spatial domain is an operation defined for two

Figure 18.15 2-D Convolution

2-D discrete functions x[m, n] and h[m, n]:

y[m, n] = ∑_{i=−k}^{k} ∑_{j=−k}^{k} x[m − i, n − j] h[i, j]  (18.52)

where h[m, n] is called the convolution kernel which typically is assumed to be central symmetric h[−m, −n] = h[m, n]. Such a convolution is sometimes also considered as a filters, by which a 2-D signal x[m, n], typically a digital image, is filtered in some way (low-pass, high-pass, band-pass, etc.) to become y[m, n]. Fig. 18.15 illustrates a 2-D convolution with a 3 × 3 kernel.

We see that each pixel y[m, n] of the resulting image y is the sum of 9 products. As the kernel slides over the input image x pixel by pixel and then row by row, all pixels of y are calculated.

For example, the four kernels below are can be used to detect edges in vertical and horizontal orientations:

[ \begin{matrix}−1 & 0 & 1 \\ −1 & 0 & 1 \\ −1 & 0 & 1 \end{matrix} ] , [ \begin{matrix}1 & 1 & 1 \\ 0 & 0 & 0 \\ −1 & −1 & −1 \end{matrix} ] , [ \begin{matrix}1 & 0 & −1 \\ 1 & 0 & −1 \\ 1 & 0 & −1 \end{matrix} ] , [ \begin{matrix}−1 & −1 & −1 \\ 0 & 0 & 0 \\ 1 & 1 & 1 \end{matrix} ]

(18.53)

and the following four kernels are for detecting edges in two diagonal orientations:

[ \begin{matrix}0 & 1 & 1 \\ −1 & 0 & 1 \\ −1 & −1 & 0 \end{matrix} ] , [ \begin{matrix}1 & 1 & 0 \\ 1 & 0 & −1 \\ 0 & −1 & −1 \end{matrix} ] [ \begin{matrix}0 & −1 & −1 \\ 1 & 0 & −1 \\ 1 & 1 & 0 \end{matrix} ] , [ \begin{matrix}−1 & −1 & 0 \\ −1 & 0 & 1 \\ 0 & 1 & 1 \end{matrix} ]

(18.54)

Fig. 18.16 shows the effects of edge detection by convolution. The cat image is

Figure 18.16 Directional Filtering

convolved with the kernels for in Eq. (18.53), while the panda image is convolved with kernels in Eq. (18.54).

We note that the convolution as a linear combination can be described by the general neural network operation y = f(x, w) with the elements in the 2-D kernel w treated as the synaptic weights, through which the 2-D output x from a lower layer is fed forward into the next layer to produce its 2-D output y.

These 3 × 3 kernels can be extended to larger kernels for detecting more complex features in a larger area of the visual field in higher layers of the network.

Specifically, the following operations take place sequentially along the multilayer pathway of the CNN hierarchy as shown in Fig. 18.17, each serving certain purposes:

• Convolution:

Each node in the convolution layers is only locally connected to a set of neighboring nodes in a square patch in the previous layer. In particular, a node in the first layer take as input the pixel values inside its receptive fields, a subregion of the image. It then generates the convolution, the sum of the inputs weighted by the kernel components, and feed it to the next layer. At each position, multiple nodes with the same receptive field form a column in the depth dimension, all performing convolution but based on different kernel to extract different types of local features, such as edges in various orientations as shown above.

• Padding:

Padding is needed for the output of convolution to maintain the same size as the input. For example, for a 5 × 5 kernel, the first pixel at the top-left corner of the output is in the position of the pixel in the 3th row and 3th column in the input, i.e., the output is 4 pixels smaller in both dimension than the input. To prevent this size reduction from one layer to the next, the input needs to be padded with additional rows and columns of pixels around its four borders (zeros or taking same value as the border pixels).

• Stride:

Stride s is the number of pixels for the revolution kernel to shift in both dimensions in convolution. In regular convolution, s = 1, the size of the resulting image is the same as the input. But when s > 1, the kernel shifts s pixels from the center of one receptive field to next. In other words, stride is the distance between the receptive field centers of neighboring neurons. The size of the output will be 1/s of the original in both dimensions.

• Pooling:

Pooling is a down-sampling method by which a local patch (e.g., 2 × 2) in the previous layer is represented by their average or maximum as a single value in the next layer called pooling layer, for dimension reduction that serves two purposes: (a) local shift and rotational invariance and (b) reduction of computational complexity.

• Dilution or Dropout:

Certain fraction of nodes randomly selected in a layer is dropped by setting the corresponding weights to zero with a set probability, for the purpose of preventing overfitting, so that the network is less sensitive to certain detailed variations in the data due most likely to noise and thereby more generalizable.

• In the last stage of the CNN, all nodes in a layer are fully connected to all nodes in the previous layer, they take as input the progressively larger and more complex features extracted in the previous, and learn to selectively respond to different image objects of interest based on error backpropagation algorithm, to achieve the ultimate goal of image objects recognition.

• Weight decay:

As mentioned before, a multilayer neural network can be considered as a function y = f(x, w) for modeling the relationship between x and y, same as many supervised learning algorithms regression and classification. Similarly, including a penalty term proportional to ||w|| or its square in the objective function is an effective way to regulate the algorithm and avoid overfitting.

Problems

1. Implement the perceptron network as a linear classifier (based on both onehot and binary encoding) for 3-D data generated by the function below. Show your results in confusion matrix and the error rate.

function [X,y]=Data
  d=3;                    % number of dimensions
  K=8;                    % number of classes
  Nk(1:K)=50;             % number of samples per class
  N=sum(Nk);              % total number of samples

Figure 18.17 Hierarchical Convolutional Network

    Means=[-1 -1 -1 -1 1 1 1 1; % mean vectors of K classes
           -1 -1 1 1 -1 -1 1 1;
           -1 1 -1 1 -1 1 -1 1];
    X=[];
    y=[];
    s=0.22;
    for k=1:K               % for each of the K classes
        Xk=Means(:,k)+s*randn(d,Nk(k));
        y=[y repelem(k,Nk(k))]
        X=[X Xk];
    end
end

2. Implement kernel perceptron network (based on radial basis kernel) and apply it to the classification of the iris dataset. Use both onehot and binary encoding methods. Use half of the data for training and the other half for testing. Show your results in confusion matrix and error rate.

3. Repeat the previous problem using to the handwritten digit dataset.

4. Implement the back propagation network and apply it to the handwritten digit dataset based on all d = 256 dimensions of the data. Use both one-hot (10 output nodes each for one class) and binary encoding (4 output nodes to encode 10 classes) labeling. Use half of the data for training and the other half for testing. Show your results in confusion matrix and error rate.

Then use the KLT method based on the between-class scatter matrix S_{B} to reduce the data dimensionality while keeping 99% of the signal energy, and then repeat the classification task above.

5. Implement autoencoder for dimensionality reduction and apply it to the iris dataset. Visualize and classify the data in the resulting lower dimensional space (both 2-D and 3-D space). Then compare the separability J_{B/T} and classification error rate with that obtained based on the PCA method.

6. Repeat the problem above using the handwritten digit dataset.

19 Competition-Based Networks

19.1 Competitive Learning Network

Competitive learning is a neural network algorithm for unsupervised clustering, similar to the K-means algorithm considered previously. The competitive learning takes place in a two-layer network, composed of an input layer of d nodes that receives an input vector x = [x_{1}, ⋯, x_{d}]^{T} as a point in the d-dimensional feature space, and an output layer of K nodes that produces an output pattern y = [y_{1}, ⋯, y_{K}]^{T} representing the clusters of interest. Each input variable x_{i} may take either a continuous real value or a binary value depending on the specific application, the outputs y_{i} ∈ {0, 1} are binary, of which only one is 1 while all others are 0 (one-hot), as the result of a winner-take-all competition based on their activation values.

In each iteration of the learning process, a pattern x randomly chosen from the unlabeled dataset X = [x_{1}, ⋯, x_{N}] is presented to the input layer of the network, while each node of the output layer gets an activation value, a linear combination of all such d inputs, i.e., the inner product of its weight vector w_{i} and the input vector x:

a_{k} = ∑_{j=1}^{d} w_{ij}x_{j} = w_{k}^{T} x = ||w_{k}|| ||x|| cos θ (k = 1, ⋯, K)  (19.1)

where θ is the angle between vectors w_{k} and x in the d-dimensional feature space. If the data vectors are normalized with unit length ||x||^{2} = 1, they are points on the unit hypersphere in the space. Moreover, if the weight vectors w_{k} is also normalized with ||w_{k}|| = 1, then their inner product above is only affected by the angle θ between them. In practice, the normalization of both the data vector x and the weight vector w is optional.

The output of the network is determined by a winner-take-all competition. The node in the output layer that is maximally activated outputs 1 while all others output 0:

y_{i} = \{ \begin{matrix}1 & if a_{i} = w_{i}^{T} x = max_{j} w_{j}^{T} x \\ 0 & otherwise \end{matrix} , (i = 1, ⋯, K)  (19.2)

Figure 19.1 Modification of Weight Vector of the Winner

The inner product w_{k}^{T} x is closely related to the Euclidean distance squared:

d^{2}(w_{k}, x) = ||w_{k} − x||^{2} = (w_{k} − x)^{T} (w_{k} − x) = w_{k}^{T} w − 2w_{k}^{T} x + x^{T} x
= ||w_{k}||^{2} + ||x||^{2} − 2w_{k}^{T} x = ||w_{k}||^{2} + ||x||^{2} − 2||w_{k}|| ||x|| cos θ (19.3)

If both x and w_{k} are normalized, i.e., they becoem points on a unit sphere in the d-dimensional feature space, then the winning node with the maximum activation a_{k} = w_{k}^{T} x also has the minimum Euclidean distance ||w_{k} − x|| and angular distance θ. The competition above can therefore be expressed as the following:

y_{k} = \{ \begin{matrix}1 & if ||w_{k} − x|| = min_{j} ||w_{j} − x|| \\ 0 & otherwise \end{matrix} , (i = 1, ⋯, K)  (19.4)

Now the competitive learning rule can be expressed as below based on y_{k}:

w_{k}^{new} = w_{k}^{old} + y_{k} η (x − w_{k}^{old}) = \{ \begin{matrix}(1 − η)w_{k}^{old} + ηx & if y_{k} = 1 \\ w_{k}^{old} & if y_{k} = 0 \end{matrix}  (19.5)

i.e., only the winning node learns to respond to the current input by modifying its weight vector, while the weights of all other nodes remain the same. This competitive learning is illustrated in Fig. 19.1, where the modified weight vector of the winner w^{new} is between x and the old w^{old}, i.e., the weight vector of the winner which is closest to the current input pattern x is pulled even closer to it, so that the node will be more likely to win and respond to the same input when it is presented again to the input of the network.

In the competitive learning rule, 0 < η < 1 is the learning rate (step size), that is to be gradually reduced through the iterations from its initial value (e.g., η_{0} = 0.9) toward zero, by a decay factor α. Due to the exponential decay of the learning rate, the iteration gradually converges to a stable state. The value of the decay factor (e.g., α = 0.99) can be determined heuristically and experimentally depending on factors such as the size of the dataset.

Here are the iterative steps in the competitive learning:

• Step 0: Normalize all data vectors x, initialize randomly the weight vectors w for the output nodes (e.g., any K randomly chosen samples from the dataset): w_{1}^{(0)}, w_{2}^{(0)}, ⋯, w_{K}^{(0)}, set iteration index to zero l = 1;

• Step 1: Choose randomly an input pattern x from the dataset and calculate the activation for each of the output nodes:

a_{k} = w_{k}^{T} x (k = 1, ⋯, K)  (19.6)

• Step 2: find the index of the winning node i = arg max_{j} a_{j}, and update its weights:

w_{i}^{(l+1)} = w_{i}^{(l)} + η(x − w_{i}^{(l)})  (19.7)

Reduce learning rate η ← αη, renormalize w_{i}^{(l+1)}.

• Step 3: Terminate the iteration if the clustering result has gradually stabilized, when the learning rate η is reduced from its initial value η_{0} to some small value (e.g., 0.1) and the weight vectors no longer change significantly. Otherwise l ← l + 1, go back to Step 1.

This iterative process can be more intuitively understood as shown in Fig. 19.2. Every time a sample x (a red dot) is presented to the input layer, one of the output nodes with its weight vector w (an blue x) closest to the current input x will become the winner and its weight vector is drawn even closer to x. This process is carried out iteratively many times, each cluster of similar sample points in the space will draw one of the weight vectors towards its central area, and the corresponding output node will always win and output 1 whenever any member of the cluster is presented to the network in the future. In other words, after this unsupervised learning, the feature space is partitioned into K regions each corresponding to one of the K clusters, represented by one of the output nodes, whose weight vector is in the central area of the region. We see that this process is very similar to the K-means clustering algorithm, in which each cluster is represented by the mean vector of all of its members.

It is possible in some cases that the data samples are not distributed in such a way that they form a set of clearly separable clusters the feature space. In the extreme case, they may even form a continuum. In such cases, a small number of output nodes (possibly even just one) may become frequent winners, while others become “dead nodes” if they never win and consequently never get the chance to modify their weight vectors. To avoid such meaningless outcome, a mechanism is needed to ensure that all nodes have some chance to win. This can be implemented by including an extra bias term in the learning rule:

a_{k} = w_{k}^{T} x + b_{k}  (19.8)

where b_{k} is the bias term proportional to the difference between the “fair share”

Figure 19.2 Weight Vectors Moving toward Centers of Clusters

of winning 1/K and the actual winning frequency:

b_{k} = c (\frac{1}{K} − \frac{number of winnings of the kth node}{total number iterations so far})  (19.9)

The value of the bias term b_{k} for the kth node will change its winning frequency. If the node is winning more than its share 1/K, then b_{k} < 0 and it becomes harder for it to win in the future. On the other hand, if the node rarely wins, b_{k} > 0 and its chance to win in the future will be increased. Here hyperparameter c is some scaling coefficient. The greater c, the more balanced the competition will be. It needs to be fine tuned based on the specific nature of the data being analyzed.

This process of competitive learning can also be viewed as a vector quantization process, by which the continuous vector space is quantized to become a set of K regions, called Voronoi diagram (tessellation). Vector quantization can be used for data compression. A cluster of similar signals x in the vector space can all be approximately represented by the weight vector w of one of a small number of K output nodes in the neighborhood of x, thereby the data size can be significantly reduced.

Same as in K-means clustering, competitive learning network also suffers from the problem that the ground truth number of clusters is typically unknown, and therefore it is difficult to determine the number of output nodes K of the network, which in the ideal case should be the same as the number of clusters in the dataset, so that each cluster can be represented by one of the nodes at the end of the clustering process. Similar to the case of K-means clustering, this issue can be addressed by carrying out the clustering process multiple times with different K values and checking to see which value best fits the data measured quantitatively by one of the criteria for the separability of the resulting cluster, such as those discussed in Section 15.1.

The Matlab code for the iteration loop of the algorithm is listed below.

[d N]=size(X);               % dataset containing N samples

b=zeros(1,K);                % bias terms for K clusters
freq=zeros(1,K);             % winning frequencies
eta=0.9;                     % initial learning rate
decay=0.99;                  % decay factor
it=0;
while eta>0.1                % main iterations
    it=it+1;                 % iteration counter
    W0=W;                    % initial weight vectors
    x=X(:,randi(N,1));       % randomly select an input sample
    dmin=inf;
    for k=1:K                % find winner among all output nodes
        d=norm(x-W(:,k))-b(k);
        if dmin>d
            dmin=d; m=k;     % mth node is the winner
        end
    end
    w=W(:,m)+eta*(x-W(:,m)); % modify winner's weights
    W(:,m)=w/norm(w);        % renormalize its weights (optional)
    share(m)=share(m)+1;     % update share for winner
    b=c*(1/K-share/it);      % modify biases for all nodes
    eta=eta*decay;           % reduce learning rate
end

Example 19.1 The competitive learning method is applied to a set of simulated 2-D data of four clusters. The network has d = 2 input nodes and m = 4 output nodes. The first 16 iterations are shown in Fig. 19.3, where open circles represent the data points, while the solid back squares represent the weight vectors. Also, the blue and red squares represent the weight vector of the winner before and after its modification, visualized by the straight line connecting the two squares. We see that the weight vectors randomly initialized are iteratively modified one at a time, and after these 16 iterations, they have each moved to the center of one of the four clusters. The separability of the clustering result measured by tr(S_{T}^{−1}S_{B}) is 1.0. When the number of output nodes is reduced from the ground truth 4 to 3 and 2, the separability is also reduced to 0.76 and 0.46, respectively. When K = 5 output nodes are used, one of the four clusters is represented by two output nodes, indicating they can be subsequently merged.

Example 19.2 Fig. 19.4 shows the clustering results of a set of data points in a 3-D space, in terms of the weight vectors before (top) and after (bottom) of the competitive learning. The left column shows the 2-D space spanned by the first two principal components, while the right column shows the weight vectors in the original 3-D space. Again we see that the weight vectors move from their random initial positions to the centers of the eight clusters as the result of the clustering.

Figure 19.3 Iterative Learning of Competitive Network (2-D)

Example 19.3 The competitive learning algorithm is also applied to the handwritten digit dataset, and the clustering result is shown as the average of all samples in each cluster in 16 × 16 image form shown in Fig. 19.5.

19.2 Self-Organizing Map (SOM)

The Self-organizing map (SOM) is a two-layer unsupervised neural network learning algorithm that maps any input pattern presented to its input layer, a vector x = [x_{1}, ⋯, x_{d}]^{T} in a d-dimensional feature space, to a set of output nodes that forms a low-dimensional space called feature map, typically a 2-D grid (lattice), although 1-D and 3-D spaces can also be used. This is shown in Fig. 19.6. A SOM can therefore be considered as a dimensionality reduction method, by which the dataset in the high-dimensional feature space can be visualized and the features and structures of the dataset can be discovered.

The learning rule of the SOM is similar to that of the competitive learning network considered previously in that all output nodes compete based on their activation levels upon the input pattern presented to input nodes of the network.

Figure 19.4 Example of Competitive Learning (3-D)

Figure 19.5 Clustering of Handwritten Digits by Competitive Learning

However, different from the competitive learning network where only the winning node gets to modify its weight vector and learn to respond to a cluster of similar patterns individually and independently, here in SOM network all output nodes in the neighborhood of the winning node get to modify their weight vectors and thereby learn to respond collectively to a cluster of similar input patterns. As the result, all output nodes are locked into a continuum feature map with smooth and gradual variation in responsive selectivity.

The SOM as a learning algorithm is motivated by the topographic organization and and columnar structures in the brain, in which signals from various visual or auditory sensory systems are topographically projected (mapped) to

Figure 19.6 Self-Organizing Map

the corresponding cortical areas, where neighboring neurons in a local cortical area respond selectively to similar stimuli.

• Retinotopic map: Spatially adjacent stimuli in the visual field are represented and responded to by neurons in adjacent positions in the primary visual cortex (V1).

• Tonotopic map: Sound signals of different frequencies are mapped to the primary auditory cortex where neighboring neurons respond to similar frequencies.

• Columnar architecture: Visual signals for different edge orientations and different motion directions are mapped respectively to the primary visual area (V1) and the middle temporal area (V5 or MT), where neurons in the same column respond to the same edge orientations or motion similar directions, and neurons in neighboring columns respond to similar edge orientations or motion directions.

In a typical SOM network, the output nodes are organized in a 2-D feature map, and the competitive learning algorithm previously discussed is modified so that the learning takes place at not only the winning node in the output layer, but also a set of nodes in the neighborhood of the winner. The weight vectors of all such nodes are modified:

w_{i}^{new} = w_{i}^{old} + u_{i}η (x − w_{i}^{old})  (19.10)

where u_{i} is a weighting function based on the Euclidean distance d_{i} from the ith node in the neighborhood to the winner in the 2-D feature map, such as a Gaussian function centered at the winner:

u_{i} = exp (−\frac{d_{i}^{2}}{2σ^{2}} ) \{ \begin{matrix}= 1 & for winner with d_{i} = 0 \\ < 1 & for all other nodes \end{matrix}  (19.11)

where σ is a hyperparameter that controls the width of the Gaussian function. We see that all neurons in the neighborhood of the winner learn from the current input pattern in such a way that their weight vectors are modified to be closer to the input vector in the d-dimensional feature space, but they do so to different extents. Specially, for the winner itself with d_{i} = 0 and u_{k} = 1, the learning rate for the winner is maximally η, while all other nodes with d_{i} > 0 also modify their weights but with some lower rates u_{i}η. Those closer to the winner with smaller d_{i} and thereby greater u_{i} will learn more than those farther away.

Here are the specific steps of the SOM learning algorithm:

1. Initialize weights of a set of output nodes arranged in a 2-D map.

2. Choose randomly an input pattern vector x in the high-dimensional feature space and calculate the activation y_{i} = w_{i}^{T} x for each of the output nodes.

3. Find the winning node with maximum activation y_{k} and update the weights of all nodes in its neighborhood by Eq. (19.10).

4. Go back to step 2 until the feature map is no longer changing or a set maximum number of iterations is reached.

In the training process, both σ controlling the size of the neighborhood and η for the learning rate will be gradually reduced.

The Matlab code segment listed below is for the essential part of the SOM algorithm.

[d N]=size(X);             % dimension and number of samples
W=rand(d,M,M);             % initialization of weights
for i=1:M
    for j=1:M
        w=reshape(W(:,i,j),[d 1]);
        W(i,j,:)=W(:,i,j)/norm(w); % normalize all weight vectors
    end
end
eta=0.9;                   % initial learning rate
sgm=M;                     % width of Gaussian neighborhood
decay=0.999;               % rate of decay
for it=1:nt                % training iterations
    n=randi([1 N]);
    x=X(:,n);              % pick randomly an input
    amax=-inf;
    for i=1:M              % find the winner
        for j=1:M
            w=reshape(W(:,i,j),[d 1]);
            a=x'*w;        % activation as inner product
            if a>amax
                amax=a; wi=i; wj=j; % record winner so far

            end
        end
    end
    for i=1:M
        for j=1:M
            dist=(wi-i)^2+(wj-j)^2; % distance to winner
            c=exp(-dist/sgm);       % Gaussian weights
            w=reshape(W(:,i,j),[d 1]); % get a weight vector
            w=w+eta*c*(x-w);  % modify weights
            w=w/norm(w);      % normalize weight vectors
            W(:,i,j)=w;       % save modified weight vector
        end
    end
    eta=eta*decay;            % reduce learning rate
    sgm=sgm*decay;            % reduce width of Gaussian
end

Example 19.4 This example of “self-organizing map” is how the SOM algorithm got its name. The output nodes organized into a 2-D map learn to respond to the positions of a set of random points x = [x_{1}, x_{2}]^{T} in a d = 2 dimensional feature space. The weights of the output nodes are randomly initialized, and then modified iteratively during the learning process when the data points are repeatedly presented to the input of the network. Once the SOM has been trained, the output nodes close to each other in the 2-D feature map respond to a set of adjacent data points in the 2-D space, thereby forming a self-organized spatial map, a roughly regular grid.

The 2-D map of the output nodes is visualized as shown in Fig. 19.7, where the spatial position of each node is determined by the two components of its weight vector w = [w_{1}, w_{2}]^{T} treated as its coordinates, and each nodeis connected to its four neighbors in the 2-D grid to indicate the spatial structure of the feature map.

In this example, the competitive learning is based on the distance between the weight vector w_{i} and the current input x, instead of their inner product w_{i}^{T} x. The node with the shortest distance ||w_{i} − x|| becomes the winner, and its weight vector w is pulled even closer to the input, dragging along with it all its neighboring nodes.

The iterative modification of the weights is shown in Figs. 19.7 for the first 16 iterations and 19.8 for the subsequent iterations (in every 50 iterations). In each of the panels, the output node (blue dot) closest to the current input vector (red dot) becomes the winner and pulls the weight vectors of all neighboring nodes toward the input. This process converges to a configuration in which the output nodes form roughly a regular 2-D grid, a self-organized map in the 2-D feature space.

466  Competition-Based Networks

Figure 19.7 Iterative Learning of SOM (First 16 Iterations) Figure 19.8 Iterative Learning of SOM (Subsequent Iterations)

Figure 19.9 SOM Applied to the TSP (First 16 Iterations)

Example 19.5

In this example the SOM algorithm is applied to address the traveling salesman problem (TSP). The SOM learns to respond to the locations of a set of 25 North America cities represented by their coordinates (longitudes and latitudes) and thereby approximate the shortest path through all these cities. The initial locations of the weight vector are arranged as a 1-D loop around the cities. And the nodes of the SOM gradually learn to modify their weights based on the city locations, so that they move toward the cities nearby to eventually form a reasonably short path through all these cities. The final result is unaffected by how the weights are initialized.

This learning process is illustrated in Figs. 19.9 for the first 16 iteration and 19.10 for the subsequent iterations (in every 50 iterations), where the output nodes are represented by their weight vectors as points along the curve in the 2-D map, together with the locations of the cities as the input to the SOM one at a time.

The resulting configuration of the output nodes is not guaranteed to be the desired shortest path through all cities, but it is should be reasonably close to such a solution.

Example 19.6 In this example, a SOM network learns to respond to different colors, as points in the color space spanned by the three primary colors red, green

Figure 19.10 SOM Applied to the TSP (Subsequent Iterations)

and blue (RGB). Each node in the 2-D array of the output layer is connected to the three input nodes taking a 3-D vector for a color sample x, and generates the inner product w^{T} x of the weight vector w and the input vector x as the activation. During training, the points in a 3-D regular grid of the color space are randomly selected as the training samples, and the weights of the SOM are gradually modified so that eventually each output node learns to respond most strongly to a specific color. Here all training vectors are normalized so that they have unity intensity, but their hue and saturation are determined by the ratios of the three primaries. Similarly, the weight vectors are also renormalized after each iteration in training. As ||w|| = ||x|| = 1, the activation w^{T} x = ||w|| ||x|| cos θ = cos θ is maximized if the angle between θ between w and x is minimized, i.e., the weight vector matches the input color vector. Now the favored color of each output node can be shown by color coding the components of its weight vector as RGB, and the SOM can be visualized by the favored colors of all of its output nodes, as shown in Fig. 19.11. Here the left panel shows the SOM when all weights are randomly initialized, and middle panel shows the fully trained SOM with neighboring nodes responding to similar colors. The right panel further shows all the winners during the learning process, encoded by the input colors they win, while all other nodes that never win (but still learn to respond to certain colors along with their neighboring winners) are colored black. We see that the winners seem to form some structural patterns.

When the size of the SOM is progressively increased, more sophisticated web-

Figure 19.11 Colors Learned by a SOM

Figure 19.12 Emergent Patterns in SOM (left) and Perspective Projection of the Color Grid (right)

like patterns start to emerge, as shown in the left panel of Fig. 19.12 for a SOM of size 1000×1000. Such emergent patterns closely resemble the perspective projection of a regular grid in the 3-D color space onto an arbitrary 2-D plane, as shown in the right panel for a zoomed-in view of such a projection. We therefore see that the SOM network actually has learned to mimic the perspective projection of the training samples in the 3-D grid in the color space onto the 2-D output array. However, in an orthographic projection of the 3-D space as shown in Fig. 19.13 no such patterns emerge. An interesting question is why the SOM learns to mimic the perspective projection instead of a orthographic projection, while there is no specific mechanism that distinguish these two types of projections in the learning process.

Problems

1. Implement competitive learning network for clustering and apply it to the iris dataset. Show the result in confusion matrix and error rate. Also visualize

Figure 19.13 Orthographic and Perspective Projections of Color Grid Orthographic (left) and perspective (right) projections of the color grid

your result in 2-D and 3-D spaces with all samples color coded according to (1) the cluster they are assinged to, and (2) their ground truth labeling.

2. Repeat the previous problem but based on the handwritten digit dataset. Also visualize the mean vectors of the K = 10 clusters converted back into 16 × 16 images to see what they actually represent.

Compare all these results to those obtained by the K-means algorithm in the homework in Chapter 15.

3. Use the SOM network to learn different colors each represented by a 3-D vector for the three primary colors red, green, and blue. When the SOM has converged, show the resulting SOM by color (RGB) coding the output nodes in the 2-D array by their three weights, same as in Example 19.6.

If interested, further show the SOM map but containing only those winning nodes during the training process. All other nodes that have never won during training will remain black. Try some large size SOM (e.g., 500 × 500) and large number of training samples (e.g., 10^{3}, i.e., 10 intensity levels in the range between 0 and 255 for each of the primary colors). Observe to see if any patterns show up in the map.

4. Use the SOM network to solve the traveling salesman problem for the major North American cities listed below together with their latitude and longitude. (More cities can be found at: ttp://www.infoplease.com/ipa/A0001796.html.)

Lat.  Long.
City  minute second minute second
LosAngeles  34  3  118  15
SanF rancisco  37  47  122  26
NewY ork  40  47  73  58
Chicago  41  50  87  37
Boston  42  21  71  5
W ashington, D.C.  38  53  77  02
Miami  25  46  80  12
Minneapolis  44  59  93  14
NewOrleans  29  57  90  4
Seattle  47  37  122  20
Dallas  32  46  96  46
Cleveland  41  28  81  37
SaltLakeCity  40  46  111  54
Atlanta  33  45  84  23
Denver  39  45  105  0
KansasCity  39  6  94  35
P hoenix  33  29  112  4

Part VI Reinforcement Learning

Part VI contains only one chapter to discuss the reinforcement learning (RL), which is considered as one of the three basic machine learning paradigms, alongside supervised learning (e.g., regression in Part II and classification in Part IV), and unsupervised learning (e.g., clustering in Chapter 15). The goal of RL is for the algorithm, a software agent, to learn to make a sequence of decisions, called policy, in terms of the best actions to take for a specific task in a given environment, for the purpose of receiving the maximum cumulative rewards from the environment.

Unlike supervised learning for either regression or classification based on the training set composed of labeled data samples, RL is a learning method without any examples of optimal behaviors. Instead, RL learns on its own without relying on any labeled training data informing the agent explicitly what the correct responses are. On the other hand, Unlike unsupervised methods, which learns passively from the given dataset, RL is a trial-and-error learning method, in which the agent actively interacts with and learns from the given environment.

RL as a sequential method in the sense that the learning depends on the dynamics of the environment, which is modeled by a Markov decision process (MDP), a stochastic system of multiple states with probabilistic state transitions and rewards. Starting from an initial state, the agent goes through a sequence of states to eventually reach a destination state. If the MDP model of the environment is known in terms of the state transition and reward probabilities, then the agent only needs to determine the optimal policy in terms of what action to take at each state in the sequence to transit to the next, so that the accumulated rewards can be maximized as the consequence of its actions. The task in this model-based case is called planning.

However, if the MDP of the environment is unknown as in the case of most real world problems, the agent needs to learn the MDP environment by repeatedly running the dynamic process while following some initial policy, and gradually evaluate the received rewards and improve the policy to eventually reach optimality. Such a process of learning the environment by interacting with it is called a Monte Carlo (MC) method and the task in this model-free case is called control.

Depending on the action a taken by the agent in the state s it is current in, the system transits to the next state s′, one of the states available at s, according to a transition probability distribution p(s′|s, a). The goal of RL is to choose the optimal action to maximize the long-term expected cumulative reward. This is typically realized by the method of dynamic programming (DP), a class of algorithms that seeks to simplify a complex problem by breaking it up into a set of sub-problems to be solved recursively. The process is illustrated in Fig. 19.14 showing the t-th step of the process, where the agent makes a decision in terms of what action a_{t} to take in the current state s_{t} = s by following certain policy π(a|s), based on which the environment makes the corresponding transition to the next state s_{t+1} = s′ according to the transition probability p(s′|s, a) conditioned on the current state s_{t} = s and the agent’s action a_{t}, and

Figure 19.14 Reinforcement Learning: Agent and Its Environment

provides a reward r_{t+1}. We will consider the specific details of the DP algorithms in the subsequent sections.

20 Introduction to Reinforcement Learning

20.1 Markov Decision Process

20.1.1 Markov Chain

The mathematical framework for reinforcement learning takes the form of a Markov decision process (MDP), based on the more basic concept of a Markov process or Markov chain, which is a stochastic model of a system that can be characterized by two types of parameters: (a) the set S = {s^{1}, ⋯, s^{N}} containing all possible states of the system, and (b) the transition probability P_{mn} = P (s^{n}|s^{m}) that models the dynamics of the system in terms of its state transition from the current state s_{t} = s^{m} to the next state s_{t+1} = s^{n} for all m, n = 1, ⋯, N. Here s^{n} denotes the nth state out of all N = |S| states of the system (not s to the nth power), while s_{t} denotes the state at the t-th step of the process.

The system is assumed to be memoryless, i.e., its future depends on the current state s_{t} but is independent of the past history in terms of all previous states:

P (s_{t+1}|s_{t}, s_{t−1}, ⋯, s_{0}) = P (s_{t+1}|s_{t})  (20.1)

For example, the motion of a helicopter can be modeled by a Markov process if its state is described by its linear and angular position and velocity, as its future position and velocity are completely determined by its present state. However it cannot be modeled by a Markov process, if the state is only described by its current position, while its velocity depends also on its positions in the past.

All transition probabilities can be organized as an N × N state transition matrix:

P = [ \begin{matrix}P_{11} & ⋯ & P_{1N} \\ ⋮ & ... & ⋮ \\ P_{N1} & ⋯ & P_{NN} \end{matrix} ]  (20.2)

where P (s^{n}|s^{m}) is the probability of the transition from state s_{m} to the state s_{n}. Since at any state s_{m}, the next state has to be one of the N possible states, we have:

∑_{n=1}^{N} P_{mn} = ∑_{n=1}^{N} P (s^{n}|s^{m}) = 1,  (m = 1, ⋯, N)  (20.3)

Any matrix satisfying this property is called a stochastic matrix. In the following

Figure 20.1 Stochastic matrix for LA Weather

discussion, we will concentrate mostly on the general transition from the current state s_{t} = s to the next state s_{t+1} = s′ with transition probability denoted by P (s_{t+1} = s′|s_{t} = s) = P_{ss′}. For example, the transitions of the weather in Los Angeles (a lot of fine days!) are shown in Fig. (20.1), with the stochastic matrix shown in Eq. (20.4).

P = [ \begin{matrix}0.7 & 0.2 & 0.1 \\ 0.4 & 0.3 & 0.3 \\ 0.5 & 0.2 & 0.3 \end{matrix} ]  (20.4)

20.1.2 Markov Reward Process

A Markov reward process (MRP) is an extension of the Markov chain that also includes a reward associated with each of the states. The goal is for an agent to collect the maximum amount of rewards while going through the states of a dynamic process of the system. An MRP can be described by four sets of parameters, including the set of all rewards R, a discount factor γ, as well as the set of states S, and the state transition probabilities represented by P for a Markov chain. An MRP can therefore be represented by a quadruple (4-tuple) ⟨S, P, R, γ⟩.

Here the reward R is a feedback signal to the agent at each time step of the sequence of state transitions, indicating how well it is doing with respect to the overall task. The reward can be negative, as a penalty, for situations to be avoided. The performance of the agent is measured by the sum of rewards accumulated over all time steps of the Markov process, weighted by the discount factor γ ∈ [0, 1] that discounts future rewards. If γ is close to 0, the immediate reward is emphasized, i.e., the policy is short-sighted or greedy; while if γ is close to 1, the rewards in the future steps will be almost as valuable as immediate ones, i.e., the policy is far-sighted.

The sequence of state transitions of an MRP from an initial state s_{0} to a terminal state s_{T} is called an episode, and the number of time steps T taken by the agent is called the horizon, which can be either finite or infinite.

Both the reward r and the next state s′ at any given current state s are random variables with joint probability p(s′, r|s) = P (s_{t+1} = s′, r_{t+1} = r|s_{t} = s), based on which both the state transition and reward probabilities can be found by marginalizing r and s′ respectively:

p(s′|s) = P (s_{t+1} = s′|s_{t} = s) = ∑_{r∈R} p(s′, r|s), P (r|s) = ∑_{s′∈S} p(s′, r|s)

(20.5)

Conventionally the reward r received after arriving at state s_{t} is denoted by r_{t+1}, instead of r_{t}. The index s′ ∈ S for the summation over all states will be abbreviated to s′ in the subsequent discussion. The expected reward in state s is

r(s) = E[r_{r+1}|s_{t} = s] = ∑_{r} rP (r|s) = ∑_{r} r ∑_{s′} P (s′, r|s)  (20.6)

The return G_{t} in step s = s_{t} is defined as the accumulated reward, the sum of the immediate reward after arriving at the current state s = s_{t} and all delayed rewards in the future states up to and including reward r_{T} at the terminal state s_{T} at the end of the episode, discounted by γ:

G_{t} = r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ + γ^{T −t−1}r_{T} = ∑_{k=0}^{T −t−1} γ^{k}r_{t+k+1}  (20.7)

The state value function v(s) in state s = s_{t} is defined as the expected return E[G_{t}], which can be expressed recursively in terms of the values v(s′) of all possible next states s′ = s_{t+1}:

v(s) = E[ G_{t}|s_{t} = s ] = E[r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ |s_{t} = s]
= E[r_{t+1} + γ(r_{t+2} + γr_{t+3} + ⋯)|s_{t} = s]
= E[r_{t+1}|s_{t} = s] + γE[G_{t+1}|s_{t+1} = s′, s_{t} = s]
= r(s) + γ ∑_{s′} P (s′|s)v(s′)  (20.8)

where r(s) is given in Eq. (20.6). The value of the terminal state s_{T} at the end of an episode is zero, as there will be no next state and therefore no more future reward.

This equation is called the Bellman equation, by which the value v(s) at current state s is expressed recursively in terms of the immediate reward r(s) and the values v(s′) of all possible next states, without explicitly invoking all future rewards. Based on such a bootstrapping approach, the multi-step MRP problem is expressed recursively as a single-step subproblem concerning only one state transition from s to s′, so that the current state value can be found from all future ones by induction. For this reason, the Bellman equation plays an essential role in all reinforcement learning algorithms to be considered below.

The Bellman equation in Eq. (20.8) holds for all N states s^{1}, ⋯, s^{N} ∈ S, and the resulting N equations can be expressed in vector form as:

v = [ v(s^{1}) ; ⋮ ; v(s^{N}) ] = [ r(s^{1}) ; ⋮ ; r(s^{N}) ] + γ [ \begin{matrix}P (s^{1}|s^{1}) & ⋯ & P (s^{N}|s^{1}) \\ ⋮ & ... & ⋮ \\ P (s^{1}|s^{N}) & ⋯ & P (s^{N}|s^{N}) \end{matrix} ] [ v(s^{1}) ; ⋮ ; v(s^{N}) ]
= r + γPv  (20.9)

where P is state transition matrix previously defined in Eq. (20.2), a stochastic matrix. Now the Bellman equation can be solved to get the value functions at all N states:

v = (I − γP)^{−1}r  (20.10)

Such a solution exists because matrix I − γP is invertible. To see this we note that all eigenvalues of the stochastic matrix P are no greater than 1: |λ| ≤ 1 (Section A.3.1), and all eigenvalues of γP are smaller than 1: γλ < 1, i.e., all eigenvalues of I − γP are greater than zero: 1 − γλ > 0. Consequently the determinant det(I−γP) of this coefficient matrix, as the product of all its nonzero eigenvalues, is non-zero and therefore matrix det(I − γP) is invertible.

Alternatively, the Bellman equation in Eq. (20.8) can also be solved iteratively based on dynamic programming, which solves a multi-stage planning problem by backward induction and finds the value function recursively. We first rewrite the Bellman equation as

v = r + γPv = B(v)  (20.11)

where B(v) = r + γPv is defined as a vector-valued function (an operator) applied to the vector argument v.

We can show that B(v) is a contraction mapping first introduced in Section 1.3, that satisfies:

||B(v_{i}) − B(v_{j})|| = ||r + γPv_{i} − (r + γPv_{j}) || = γ||P(v_{i} − v_{j})||
≤ γ||P|| ||v_{i} − v_{j}|| = γ||v_{i} − v_{j}|| < ||v_{i} − v_{j}|| (20.12)

We can also prove that B(v) is a contraction mapping by showing that the norm of its Jacobian matrix is smaller than 1. We first find the Jacobian matrix of B(v), its derivative with respect to its vector argument v:

J_{B} = \frac{d}{dv} B(v) = \frac{d}{dv} (r + γBv) = γP  (20.13)

We further find its p-norm with p = ∞ (equivalent to p = 1, 2), the maximum absolute row sum:

||J_{B}||_{p=∞} = ||γP||_{∞} = γ||P||_{∞} = γ max_{1≤i≤N} ∑_{j=1}^{N} |P_{ij}| = γ < 1  (20.14)

The last equality is due to Eq. (20.3), i.e., ||P||_{∞} = 1.

Having shown that B(v) is a contraction mapping, we can now use the method of fixed-point iteration first introduced in Section 1.3 to solve the Bellman equation by iterating Eq. (20.11) from an arbitrary initial value, such as v_{0} = 0:

v_{n+1} = B(v_{n}) = r + γPv_{n}  (20.15)

This iteration will always converge to the root of the equation, the fixed point of function B(v).

In summary, the Bellman equation in Eq. (20.8) can be solved by either of the two methods in Eqs. (20.10) and (20.15), so long as γ < 1. When the size of state space, the cardinality |S| = N of the set of all states, is large, the complexity O(N^{3}) is high enough that the iterative method may be more efficient.

20.1.3 Markov Decision Process

A Markov decision process (MDP) is an extension of the MRP that also includes a certain decision-making rule called policy denoted by π. Following a given policy the agent takes an action a ∈ A(s) out of all actions available at a state s, denoted by A(s), to control the state transition of the dynamic process in the system. An MDP can therefore be described by a quintuple (5-tuple) < S, P, R, A, γ >, where A represents the set of all actions. A policy can be either deterministic denoted by a = π(s), or stochastic denoted by π(a|s) = P (a_{t} = a|s_{t} = s) as the conditional probability of taking action a in state s. A policy is soft if any of the actions available at a state is possible to be taken, i.e., π(a|s) > 0 for all a ∈ A(s) and all s ∈ S. In particular, a policy is ϵ-soft if π(a|s) ≥ ϵ/|A(s)| for some small value of ϵ.

As there are |A(s^{n})| available actions to take in each state s^{n} ∈ S, there are in total |A(s^{1})| ⋯ |A(s^{N})| = Π_{n=1}^{N}|A(s^{n})| possible deterministic policies. If A(s) = A is the same for all |S| states, then the total number of policies is |A|^{|S|}. Following a given policy from an initial state s_{0} will result in a sequence of state transitions called a trajectory of the episode:

s_{0} ⟶^{a_{0}} s_{1} ⟶^{a_{1}} s_{2} ⟶^{a_{2}} s_{3} ⋯ ⟶^{a_{T −1}} s_{T}  (20.16)

of which the state s_{t} visited at the t-th step can be any of the N states in S. Note that in an episode some of the states in S may be visited multiple times, while some others may never be visited at all. Also note that due to the random nature of the MDP, following the same policy may not result in the same trajectory.

Along the trajectory of an episode starting from state s_{0}, the discounted rewards from all states visited will accumulate, represented by the value v(s_{0}), the expectation of the sum of all discounted future rewards. The goal of planning is to find the optimal policy as a sequence of actions for a given MDP model of the environment, so that the v(s_{0}) is maximized. This optimization problem can be solved by the method of dynamic programming.

In an MDP, both the next state s′ and reward r are random variables with joint probability P (s′, r|s, a) = P (s_{t+1} = s′, r_{t+1} = r|s_{t} = s, a_{t} = a) conditioned on the action a and the previous state s. The state transition and reward probabilities can be found by marginalization (similar to Eq. (20.5) for an MRP):

P (s′|s, a) = ∑_{r∈R} p(s′, r|s, a)
P (r|s, a) = ∑_{s′∈S} P (s′, r|s, a)  (20.17)

and the expected reward r received after arriving at the current state s is

r(s, a) = E[r_{t+1}|s_{t} = s, a_{t} = a] = ∑_{r} r P (r|s, a)  (20.18)

Following a given random policy π(a|s), the agent takes an action a ∈ A(s) to transit from the current state s_{t} = s to the next state s_{t+1} = s′ with probability

P_{π}(s′|s) = P (s_{t+1} = s′|s_{t} = s) = ∑_{a∈A(s)} π(a|s) P (s′|s, a)  (20.19)

satisfying ∑_{s′} P_{π}(s′|s) = 1, and receives a reward in state s:

r_{π}(s) = E_{π}[r_{t+1}|s_{t} = s] = ∑_{a∈A(s)} π(a|s) r(s, a)  (20.20)

where E_{π}[x] denotes the expectation (of some random variable x inside the brackets) with respect to a certain policy π(a|s). The summation over all available actions a ∈ A(s) in state s will be abbreviated to a in the following.

Consider, as a simple and typical example, the gridworld problem, where the environment is a rectangular 2-D space divided into squares, each considered as a state with certain reward. The goal for the agent in this problem is to start at a specified position (e.g., the square at the upper-left corner) and travel to a destination (e.g., the square at the lower-right corner) while trying to accumulate the maximum amount of reward along the way. There are four actions (going up, right, down, and left) available at an internal square, but only three or two at a square on the boarder or in the corner of the grid. Upon arrival at a square as a state s, the agent receives a reward r, and then makes a state transition to one of the neighboring squares as the next state s′, depending on the action a it takes while following a certain policy π. Note that both the reward r and the next state s′ can be either deterministic or probabilistic, and so is the policy π.

Moreover, certain constraints may be imposed on the problem. For example, some of the squares are not allowed to enter, such as the walls in a maze. In this case the problem becomes more challenging, not only because the available actions at some states are reduced, but also some paths may lead to a dead end if the policy is to near-sighted (not far-sighted enough).

In reinforcement learning, to prepare for the algorithms, we further define two important functions with respect to a given policy π of an MDP:

• The state value function v_{π}(s) is the expected return in state s of the MDP while following policy π:

v_{π}(s) = E_{π}[G_{t}|s_{t} = s]
= ∑_{a} π(a|s) (r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′))
= ∑_{a} π(a|s) q_{π}(s, a)  (20.21)

This is similar to the value function v(s) of an MRP in Eq. (20.8), but now treated as a function of action a as well as state s, based on q_{π}(s, a) as defined below.

• The state-action value function q_{π}(s, a) is the expected return of taking a specific action a (regardless of policy π) in state s, and then following π in all subsequent states:

q_{π}(s, a) = E_{π} [G_{t}|s_{t} = s, a_{t} = a]
= r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′)
= r(s, a) + γ ∑_{s′} P (s′|s, a) ∑_{a′} π(a′|s′) q_{π}(s′, a′)) (20.22)

where v_{π}(s′) is recursively represented as a weighted sum of q_{π}(s′, a′) based on Eq. (20.21).

The state-action value q_{π}(s, a) plays a more important role than v_{π}(s) as it allows the freedom of taking any action independent of a given policy π, thereby allowing the opportunity to improve an existing policy, e.g., by taking a greedy action to maximize the value q_{π}(s, a). The state-action value function q_{π}(s, a) is often abbreviated as the action value or simply Q-value for convenience in future discussions.

As a function of the state-action pair (s, a), the state-action value q_{π}(s, a) can be represented as a table of |S| rows, each for one of the states s, and |A| columns each for one of the actions a. The Q-value for each state-action pair stored in the table can be updated iteratively by various algorithms for learning the Q-values.

When in particular the policy is deterministic with π(a|s) = 1 and a = π(s), then we have

P_{π}(s′|s) = P (s′|s, a),  r_{π}(s) = r(s, a)  (20.23)

and the Bellman equations in Eqs. (20.21) and (20.22) become the same:

v_{π}(s) = r(s, a) + γ ∑_{s′} P (s′|s, a) v_{π}(s′) = q_{π}(s, π(s))  (20.24)

This is the same as the Bellman equation of an MRP in Eq. (20.8), which can be solved iteratively if γ < 1, just as in Eq. (20.15).

Fig. 20.2 illustrates the Bellman equations in Eqs. (20.21) and (20.22), showing the bootstrapping of the state value function in the dashed box on the left, and

Figure 20.2 Bellman Equations for State and Action Values

that of the action value function in dashed box on the right. Here the word bootstrapping means the iterative method that updates the estimated value at a state s based on the previously estimated value of the next state s′. After taking one of the actions a ∈ A(s) based on policy π(a|s), an immediate reward r(s, a) is received.

Given a policy, it is still not certain which state the MDP will transit into as the next state s′ due to the random nature of the state transition described by P (s′|s, a). This uncertainty is represented symbolically by a dashed circle called a chance node associated with the action value q_{π}(s, a) as the sum of the immediate reward r(s, a) and the expected value of the next state, the weighted average of values v(s′) of all possible state s′ ∈ S.

20.2 Model-Based Planning

Given a complete MDP model in terms of p(s′, r|s, a) of the environment, the goal of model-based planning is to find the optimal policy π^{∗} that achieves the maximum value:

v^{∗}(s) = max_{π} v_{π}(s),  π^{∗}(s) = arg max_{π} v_{π}(s) ≥ π(s) ∀π  (20.25)

based on the partial order that compares different policies according to their corresponding values:

If v_{π_{1}}(s) ≥ v_{π_{2}}(s) ≥ ⋯ v_{π_{k}}(s), ∀s ∈ S
Then π_{1} ≥ π_{2} ≥ ⋯ ≥ π_{k}  (20.26)

The problem of policy optimization can be addressed based on two subproblems: policy evaluation to measure how good a policy is, and policy improvement to get a better policy with higher values.

One way to solve this optimization problem is to use brute-force search to enumerate all |A(s)| available policies at each of the |S| states s ∈ S. If we assume the numbers of available actions in all states are the same for simplicity, i.e., A(s^{1}) = ⋯ = A(s^{N}) = A, then the computational complexity of this method is O(|A|^{|S|}), likely to be too high for the brute-force search to be practical if |A| is large.

A more efficient and therefore more practical way to optimize the policy is to do it iteratively. Consider a simpler task of improving upon an existing deterministic policy π at a single transition step from the current state s to a next state s′. Instead of taking the action a = π(s) dictated by the policy, we can improve it by taking the greedy action that maximizes the action value function in Eq. (20.22), while subsequent steps still follow the old policy π. Now we get a new policy:

π′(s) = arg max_{a} q_{π}(s, a) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)] (20.27)

so that the action value is higher than that of the old one:

q_{π}(s, π′(s)) = max_{a} q_{π}(s, a) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)]
≥ q_{π}(s, π(s)) = v_{π}(s)  (20.28)

where the last equality is due to Eq. (20.24) for a deterministic policy.

While it is obvious that this greedy method will result in a higher return for the single state transition from s to s′, we still need to prove that by following this greedy method in all subsequent states, we will get a new policy π′ that achieves higher values than those of the old one π at all states, i.e.,

v_{π′}(s) ≥ v_{π}(s),  ∀s ∈ S  (20.29)

This is the policy improvement theorem, which can be proven by recursively applying the greedy method to replace the old policy π(s) by the new one π′(s) for a higher value for each of the subsequent steps one at a time:

v_{π}(s) ≤ q_{π}(s, π′(s)) = max_{a} q_{π}(s, a) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π}(s′)]
= r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s))v_{π}(s′)
≤ r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) max_{a′} q_{π}(s′, a′)

= r(s, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) [r(s′, π′(s)) + γ ∑_{s′′} P (s′′|s′, π′(s′))v_{π}(s′′)]

= r(s, π′(s)) + γr(s′, π′(s)) + γ ∑_{s′} P (s′|s, π′(s)) [γ ∑_{s′′} P (s′′|s′, π′(s′))v_{π}(s′′)]
≤ ⋯
= r(s, π′(s)) + γr(s′, π′(s′)) + γ^{2}r(s′′, π′(s′′) + ⋯
= v_{π′}(s)  (20.30)

Figure 20.3 Bellman Optimality Equations for State and Action Values

i.e., π′(s) ≥ π(s).

(This derivation may seem long and difficult to follow (although each step should be straightforward), a less experienced reader may choose to skip it and move on when studying the topic the first time.)

The optimal state value v^{∗}(s) and action value q^{∗}(s, a) can therefore be achieved by replacing the weighted average over all possible actions at each state (s and s′) in the Bellman equations in Eqs. (20.21) and (20.22) by their maximum:

v^{∗}(s) = max_{a} q^{∗}(s, a) = max_{a} (r(s, a) + γ ∑_{s′} P (s′|s, a)v^{∗}(s′))  (20.31)

and

q^{∗}(s, a) = r(s, a) + γ ∑_{s′} P (s′|s, a)v^{∗}(s′) = r(s, a) + γ ∑_{s′} P (s′|s, a) max_{a′} q^{∗}(s′, a′)

(20.32)

If the state transition is deterministic, this equation becomes

q^{∗}(s, a) = r(s, a) + γ max_{a′} q^{∗}(s′, a′)  (20.33)

Similar to Eqs. (20.21) and (20.22) illustrated in Fig. 20.2, Eqs. (20.31) and (20.32), called Bellman optimality equations, are illustrated in Fig. 20.3.

The Bellman optimality equations should hold for all N states in S = {s^{1}, ⋯, s^{N}} and can therefore be written in vector form:

v^{∗} = max_{a} (r_{a} + γP_{a}v^{∗})  (20.34)

where

v^{∗} = [ v^{∗}(s^{1}) ; ⋮ ; v^{∗}(s^{N}) ] , r_{a} = [ r(s^{1}, a) ; ⋮ ; r(s^{N}, a) ]

P_{a} = [ \begin{matrix}P (s^{1}|s^{1}, a) & ⋯ & P (s^{N}|s^{1}, a) \\ ⋮ & ... & ⋮ \\ P (s^{1}|s^{N}, a) & ⋯ & P (s^{N}|s^{N}, a) \end{matrix} ]  (20.35)

This nonlinear equation system can be solved iteratively for v^{∗} (similar to Eq. (20.15):

v_{n+1}^{∗} = max_{a} (r_{a} + γP_{a}v_{n}^{∗}) = B(v_{n})  (20.36)

This iteration will converge, as function B(v) is a contraction mapping with γ < 1:

||B(v_{i}) − B(v_{j})|| = || max_{a_{i}} (r_{a_{i}} + γP_{a_{i}}v_{i}) − max_{a_{j}} (r_{a_{j}} + γP_{a_{j}}v_{j}) ||
≤ || max_{a_{i}} (r_{a_{i}} + γP_{a_{i}}v_{i} − r_{a_{i}} − γP_{a_{i}}v_{j}) ||
= max_{a_{i}} γ||P_{a_{i}}(v_{i} − v_{j})|| ≤ max_{a_{i}} γ||P_{a_{i}}|| ||v_{i} − v_{j}||
= γ||v_{i} − v_{j}|| < ||v_{i} − v_{j}||  (20.37)

where ||P|| = ||P||_{∞} = 1 is the p-norm (p = ∞) of a stochastic matrix, which is known to be 1.

This iteration will be used for policy evaluation in the algorithm value iteration to be considered below for finding the optimal policy.

Given the complete information of the MDP model in terms of p(s′, r|s, a), we can find the optimal policy π^{∗} using either policy iteration (PI) or value iteration (VI) based on the method of dynamic programming.

• Policy Iteration

Policy iteration carries out the following two intertwined and interacting processes iteratively from some arbitrary initial policy π until convergence when π no longer changes:

– Policy evaluation:

Given a deterministic policy π, Eq. (20.21) becomes the Bellman equation in Eq. (20.24), a linear equation system of N = |S| equations of N unknowns v_{π}(s^{1}), ⋯, v_{π}(s^{n}), which can be solved iteratively as in Eq. (20.15) to find values at all states: v_{n+1}^{π}(s) = r(s, π(s)) + γ ∑_{s′} P (s′|s, π(s))v_{n}^{π}(s′) ∀s ∈ S

– Policy improvement:

Given v_{π}(s) based on policy π, get a better policy π′ by the greedy method as in Eq. (20.27): π′(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, π(s))v_{π}(s′)] ∀s ∈ S

We see that in each round of the policy iteration, the processes of policy evaluation and policy improvement, are carried out alternatively, one depending on the other, and one completing before the other starts. The value functions v_{π}(s) at all states are systemically evaluated before they are updated, and the policy π(a|s) at all states is updated before it is evaluated. These two processes chase each other as a moving target iteratively and eventually converge to the optimal policy π^{∗} that achieves the maximum value v^{∗}. This approach is said to be synchronous, as illustrated here:

π_{o} ⟶^{E} v_{π_{0}} ⟶^{I} π_{1} ⟶^{E} v_{π_{1}} ⟶^{I} ⋯ ⟶^{I} π^{∗} ⟶^{E} v_{π^{∗}} = v^{∗}  (20.38)

The pseudo code for the algorithm is listed below: Initialize:

v(s) ∈ ℝ, v(s) = 0 if s is terminal node, π(s) ∈ A(s), ∀s ∈ S; Set tolerance tol (small positive value);

repeat

Policy Evaluation (find v(s) ∀s ∈ S):

repeat

δ = 0
for ∀s ∈ S:
v′(s) = r(s, π(s))+γ ∑_{s′} P (s′|s, π(s))v(s′)
δ = max(δ, |v(s) − v′(s)|)

until δ < tol Policy Improvement (by taking greedy action):

Done = T for each s ∈ S

π′(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, π(s))v(s′)]

if π′(s) ≠ π(s), Done = F

end for

until done

• Value Iteration:

One drawback of the policy iteration method is the high computational complexity of the inner iteration for policy evaluation. To speed up this process, we can terminate the inner iteration early before convergence. In the extreme case, we simply combine policy evaluation and improvement into a single step, resulting in the following two-step value iteration:

– Find the optimal values:

Based on some initial value v_{0}(s), solve the Bellman optimality equation in Eq. (20.31) by iteration in Eq. (20.36):

v_{n+1}(s) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{n}(s′)] ∀s ∈ S (20.39)

Figure 20.4 Convergence of Policy Iteration

– Find the optimal policy:

Find π^{∗}(s) that maximizes v^{∗}(s):

π^{∗}(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v_{n}(s′)] ∀s ∈ S

(20.40)

Here is the pseudo code for the algorithm: Initialize: v(s) = 0 ∀s ∈ S repeat

δ = 0;

for each s ∈ S

v′(s) = max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v(s′)]
δ = max(δ, |v′(s) − v(s)|)
v(s) = v′(s)

end for until δ < tol π^{∗}(s) = arg max_{a} [r(s, a) + γ ∑_{s′} P (s′|s, a)v(s′)]

This PI method as illustrated in Fig. (20.4).

20.3 Model-Free Evaluation and Control

The previously discussed dynamic programming methods find the optimal policy based on the assumption that the MDP model of the environment is completely available, i.e., the dynamics of MDP in terms of its state transition and reward mechanism are known. A given policy π can then be evaluated based on the transition probabilities p(s′|s, a), and improved based on greedy action selection at each state. This model-based planning is an optimization problem involving no learning of the environment.

Here we consider the optimization problem with an unknown MDP model of the environment. In this model-free problem, called a control problem, both the state transition and reward probabilities are unknown, and the value functions can no longer be calculated directly based on Eqs. (20.21) and (20.22) as in model-based planning. Now the dynamics of the system in terms of its MDP model needs to be learned based on sampling the environment by repeatedly running a large number of episodes of the stochastic process of the environment while following some given policy, and estimate the value functions as the average of the actual returns received during the sampling process. Such a method is generally referred to as the Monte Carlo (MC) method. At the same time, the given policy can also be improved to gradually approach optimality. This approach is called general policy iteration (GPI), as illustrated below, similar to the PI method for model-based planning illustrated in Eq. (20.38).

π_{o} ⟶^{E} q_{π_{0}} ⟶^{I} π_{1} ⟶^{E} q_{π_{1}} ⟶^{I} ⋯ ⟶^{I} π^{∗} ⟶^{E} q_{π^{∗}} = v^{∗}  (20.41)

While GPI in Eq. (20.41) and GI in Eq. (20.38) may look similar to each other, they are essentially different in the following aspects.

• Policy improvement is based on the state value function v_{π}(s) in model-based planning, but the action-value function q_{π}(s, a) in model-free control. This is because without a specific MDP model the state value function can no longer be calculated, while the action value function can still be estimated based on the actual rewards received while sampling the environment. Similar to how we improve the policy by taking the greedy action to achieve a higher state value in model-based planning, here we improve the policy by taking an action different from that dictated by the given policy π to achieve a higher action value based on some ϵ-greedy method to be discussed later.

• Estimating the action value by the MC method requires running a large number of episodes while sampling the environment, and the computational complexity for policy evaluation in the model-free case is much higher than that in model-based planning. To speed up this process in model-free control, the iterative policy evaluation is no longer fully carried for it to converge to the true action value of the policy. Instead, the iterative update of the action value is carried out only once, followed immediately by the next phase of policy improvement. Such an estimated action value is inevitably very noisy, but due to the much reduced complexity, we can afford to run a large number of episodes while sampling the environment.

• In model-based planning, the action value is found based on all available actions at each state (Eq. (20.22)) during policy evaluation, and we only need to exploit the greedy action at each state to find the policy that achieves the maximum action value q_{π}(s) (Eq. (20.27)) during policy improvement. However, in model-free control, we have to learn the action value q_{π}(s, a), as a function of action a ∈ A(s) and state s by sampling the environment. This estimated action value based only on some partial sample data is inevitably noisy, especially in the early stage of learning when many of the states have not been visited yet. We therefore need to explore all actions available at each state to better learn the value function, as well as to exploit the greedy action to improve the policy.

As indicated by the last point above, a proper tradeoff between the exploitation of the greedy action and the exploration of other non-greedy actions needs be made, so that the agent can be exposed to all possible state-action pairs and gradually learn the action value function while at the same time it can also improve the policy being followed. The ϵ-greedy method is just such a method. Specifically, we define ϵ ∈ [0, 1] as the probability of exploring all actions A(S) available in state s (including the greedy one), each with an equal probability ϵ/|A(s)|, and 1 − ϵ as the probability of taking only the greedy action. Therefore the overall probability of taking the greedy action, also one of the |A(s)| available actions, is 1 − ϵ + ϵ/|A(s)|. The policy can now be expressed as

π′(a|s) = \{ \begin{matrix}arg max_{a} q(s, a) & P_{exploitation} = 1 − ϵ + ϵ/|A(s)| \\ any other a ∈ A(s) & P_{exploration} = ϵ/|A(s)| \end{matrix}  (20.42)

For example, if |A(s)| = 4 and ϵ = 2/3, then the probability of taking the greedy action is 1 − ϵ + ϵ/|A(s)| = 3/6 = 1/2, and the probability of taking any of the remaining three non-greedy actions is ϵ/|A(s)| = 1/6. Such a policy is different from the original policy currently followed by the agent. Such a policy is called behavior policy, as it describes the behavior of the agent.

The value of ϵ may vary during the process. A larger ϵ (close to 1) can be used to allow more state-action pairs to be explored at the early stage when many of them have not been visited yet, while smaller ϵ (approaching 0) may be used to exploit the greedy actions for higher values in the later stage when the action value function has been more reliably learned. For example, we can let ϵ_{k} = 1/k with k being the number of iterations completed so far, so that it’s value gets progressively smaller from a large initial value. As k approaches infinity and ϵ approaches zero, the ϵ-greedy method approaches absolute greedy method towards the end of the process.

Recall we proved the policy improvement theorem in Eq. (20.30) stating that the state value v_{π′}(s) achieved by the greedy policy π′(s) is no lower than v_{π}(s) achieved by the original policy π, i.e., π′(s) ≥ π(s). Similarly here we can also prove the same policy improvement theorem stating that the action value q_{π′}(s, a) achieved by the ϵ-greedy policy π′ is no lower than q_{π}(s, a) achieved by the original policy π:

q_{π}(s, π′(s)) = ∑_{a} π′(a|s)q_{π}(s, a)

= \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + (1 − ϵ) max_{a} q_{π}(s, a)
≥ \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + (1 − ϵ) ∑_{a} \frac{π(a|s) − ϵ/|A(s)|}{1 − ϵ}q_{π}(s, a) (20.43)

where the inequality is due to the fact that the maximum value of q_{π}(s, a) is no less than the average of action values q_{π}(s, a) over all a ∈ A(s) weighted by some normalized coefficients adding up to 1:

∑_{a} \frac{π(a|s) − ϵ/|A(s)|}{1 − ϵ} = \frac{1}{1 − ϵ} ∑_{a} (π(a|s) − \frac{ϵ}{|A(s)|}) = \frac{1}{1 − ϵ}(1 − ϵ) = 1

(20.44)

Continuing the equation above we further get

q_{π}(s, π′(s)) ≥ \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) − \frac{ϵ}{|A(s)|} ∑_{a} q_{π}(s, a) + ∑_{a} π(a|s)q_{π}(s, a)
= ∑_{a} π(a|s)q_{π}(s, a) = v_{π}(s)  (20.45)

The iterative process of the GPI method for model-free control is illustrated in Fig. 20.5, which is similar to the PI method for model-based planning based planning illustrated in Fig. 20.4, but here the state value v_{π}(s) is replaced by the action value q_{π}(s, a), and the greedy method is replaced by the ϵ-greedy method.

Comparing the methods of PI and GPI, we note that PI is a synchronous method in the sense that the value functions at all states are evaluated before the policy is updated, while GPI is an asynchronous method, in which the policy evaluation and improvement are carried out in-place for only one state at a time before all other states, and the two alternating processes of policy evaluation and policy improvement are carried out iteratively.

Also, in GPI the estimated action value q̃ is obtained with only one iteration of policy evaluation, instead of a more accurate estimate that would otherwise be obtained if the iterative evaluation were fully carried out to its convergence, represented by the top line in Fig. 20.5. At the same the time, the ϵ-greedy method is carried out based on the estimated q̃ to improvement the policy, represented by the bottom line in the figure.

The dynamic process of the environment is sampled by running a large number of episodes (all assumed to terminate) of the process while taking action a_{t} = π(s_{t}) in each state s_{t}, (t = 1, ⋯, T):

s_{0} → (a_{1}, r_{1}, s_{1}) → (a_{2}, r_{2}, s_{2}) → ⋯ → (a_{T} , r_{T} , s_{T})  (20.46)

and estimating the value functions based on the averaged returns G_{t} actually received from these episodes. The trajectory of each episode is also illustrated in Fig. 20.6.

Figure 20.5 Convergence of General Policy Iteration

Figure 20.6 A Trajectory Segment of GPI

The trajectories of different episodes are different from each other in general due to the random nature of the environment. In the following sections we will consider specific algorithms for the implementation of the general model-free control discussed above.

To prepare for the discussion of some specific GPI methods, we first consider a general problem of estimating the value of a random variable x as the running average x̂_{n} of a set of n samples x_{1}, ⋯, x_{n} collected in real time, which is updated incrementally upon receiving a new sample x_{n+1}:

x̂_{n+1} = \frac{1}{n + 1} ∑_{i=1}^{n+1} x_{i} = \frac{1}{n + 1} (x_{n+1} + ∑_{i=1}^{n} x_{i})
= \frac{1}{n + 1}(x_{n+1} + nx̂_{n}) = x̂_{n} + \frac{1}{n + 1}(x_{n+1} − x̂_{n})
= x̂_{n} + α(x_{n+1} − x̂_{n})  (20.47)

where x̂_{n} and x̂_{n+1} are respectively the old and new estimates of variable x, and α = 1/(n + 1) is the step size. If we generalize α for it to take any value between 0 and 1, then Eq. (20.47) above can be rewritten as below for three special cases for α = 0, 1/(n + 1), 1:

x̂_{n+1} = x̂_{n} + α(x_{n+1} − x̂_{n})
= \{ \begin{matrix}x_{n+1} & α = 1 & no contribution from previous samples \\ \frac{1}{n+1} ∑_{i=1}^{n+1} x_{i} & α = 1/(n + 1) & equal contribution from all samples \\ x̂_{n} & α = 0 & no contribution from latest sample \end{matrix} (20.48)

We see that by adjusting α, we can control how much the latest and previous samples contribute to the to estimated average. A reasonable strategy is therefore to progressively decrease α from its initial value 1 down to 0, so that the estimated average is mostly determined by the recent samples in the early stage of the process but it becomes progressively more stabilized when enough samples have been collected in the later stage. Such a strategy is suitable in general in estimating model parameters of a non-stationary system, such as an MDP with varying behaviors when the policy is being modified continuously.

The updating process in Eq. (20.47) can be considered as one iterative step in the method of stochastic gradient descent (SGD) (Section 6.1) for minimizing the squared error ε = (x_{n+1} − x̂_{n})^{2}/2 between the old average x̂_{n} and the latest sample x_{n+1}, and the second term is the gradient dε/dx̂_{n} weighted by the step size.

In the specific case of model-free control, Eq. (20.47) is used to iteratively update the estimated state value functions while sampling the environment. As we will see in the following sections, the state-action values are estimated as the average of all previous sample returns, and they are updated incrementally when a new sample return G_{n+1}, the target, becomes available:

v_{n+1}(s) ⇐ v_{n}(s) + α(G_{n} − v_{n}(s))
q_{n+1}(s, a) ⇐ q_{n}(s, q) + α(G_{n} − q_{n}(s, a))  (20.49)

The result of this iterative learning process is the estimated state-action values q(s, a), stored in a 2-D array, called a Q-table, of which the rows are for all the states and the columns are for all available actions at each state. Based on the Qtable, the optimal control policy can be determined by always taking the greedy action with the maximum state-action value at each state, and the solution of the reinforcement learning problem can be found by following the sequence of optimal actions as in shown Eq. (20.16) from the given initial state s_{0}. Such methods based on the Q-table are called the tabular methods.

20.3.1 Monte Carlo (MC) Algorithms

We first consider the simple problem of evaluating an existing policy π in terms of its value function given by the Bellman equation in Eq. (20.24):

v_{π}(s) = r(s, π(s)) + γ ∑_{s′} p(s′|s, π(s))v_{π}(s′)  (20.50)

This equation for the model-based case can also be used in the model-free case based on the MC method by running multiple sample episodes in the environment. The state value v_{π}(s) is estimated as the average of the actual returns G_{t}, which in turn are calculated as the sum of discounted rewards r_{t+1}, ⋯, r_{T} from the current state s = s_{t} onward to the terminal state s_{T} found at the end of the episode (assuming a finite horizon).

This method has two versions:

• Every-visit: Returns of all visits to a state in the trajectory of an episode are counted in the estimation of the state value function.

• First-visit: only the first visit to each state in each episode is counted.

In the first method, all sample returns G_{t} from multiple visits to a state s_{t} contribute to the estimated state value v_{π}(s_{t}). But these sample returns for the state may not be independent of each other as the later ones are dependent on the previous ones, i.e., they are not i.i.d. samples and the estimate values may be biased. This problem is avoided in the second method, where only the return of the first visit to each of the states is used as an independent sample drawn from the distribution, and the estimated values are no longer biased. However, as only a fraction of all samples in the trajectory is used to estimate the state values, the cost of the unbiased estimation is its high variance, which can be reduced only by running a large number of episodes to get enough samples for a more statistically reliable average. We realize that this is just the common issue of bias-variance tradeoff (Section 4.2) faced by many learning algorithms.

The pseudo code below is for first-visit version due to the if statement, which can be removed for the every-visit version.

Input: a given policy π to be evaluated Initialize: G_{t} = [ ] (empty lists) for all t = 0, ⋯, T − 1 loop (for each episode)

run episode to the end while following π to get sample data: (s_{0}, a_{0}, r_{1}), (s_{1}, a_{1}, r_{2}), ⋯, (s_{T −1}, a_{T −1}, r_{T})

G = 0

for t = T − 1, T − 2, ⋯, 0 (for each step)

G = γG + r_{t+1}
if S_{t} ∉ {S_{0}, ⋯, S_{t−1}} (first visit)
G_{t} = [G_{t}, G] (append G to list G_{t})

V (S_{t}) = average(G_{t}), ∀t = 0, , ⋯, T − 1

In this code, the function average finds the average of all elements of a list when all sample returns are available at the end of each episode. A more computationally efficient method (in terms of both space and complexity) is to find the average incrementally as in Eq. (20.49). Based on this incremental average, policy evaluation in terms of both the state and action values can also be carried out by the code below:

Initialize v(s) = 0, ∀s loop (for each episode)

run episode following π to get G_{t}, ∀t for t = 0, ⋯, T − 1 (for each step)

if s_{t} is visited the first time v(s_{t}) = v(s_{t}) + α(G_{t} − v(s_{t}))

and

Initialize: q(s, a) = 0, ∀(s, a) loop (for each episode)

run episode following π to get G_{t}, ∀t

for all (s, a) pairs visited

if (s, a) is visited the first time q(s, a) = q(s, a) + α(G_{t} − q(s, a))

The if condition can be removed for every-visit version of the algorithm.

We next consider the MC algorithm for model-free control, based on the generalized policy iteration of alternating and interacting policy evaluation and policy improvement. While sampling the environment by following an existing policy π the action value q_{π}(s), instead of the state value v_{π}(s), is gradually estimated and the policy is gradually improved at the same time. This iteration will converge to the optimal policy at the limit when the number of iterations goes to infinity.

The pseudo code for this algorithm is listed below.

Input: ϵ > 0, α > 0, and a policy π Initialize: q(s, a) = 0, ∀(s, a), π = ϵ − soft (arbitrarily) loop (for each episode)

run episode following π to get G_{0}, ⋯, G_{T −1}

for t = 0, ⋯, T − 1

if (s_{t}, a_{t}) is visited the first time q(s_{t}, a_{t}) = q(s_{t}, a_{t}) + α(G_{t} − q(s_{t}, a_{t}))

a^{∗} = arg max_{a} q(s_{t}, a_{t})
for ∀a ∈ A(s_{t})

π(a|s_{t}) = \{ \begin{matrix}1 − ϵ + ϵ/|A(s_{t})| & if a = a^{∗} \\ ϵ/|A(s_{t})| & else \end{matrix}

Here the ϵ-greedy policy π(a|s) is based on the action value function q(s, a) (maximized by the greedy action). Through out the iteration, the policy is modified following the action value function, while at the same time the action value function is modified based on the policy, until the process converges to the desired optimality.

The optimal policy π(s) obtained by the algorithm above is not completely deterministic due to the ϵ-greedy approach, but it is said to be near-deterministic as the greedy action is favored over other non-greedy actions.

20.3.2 Temporal Difference (TD) Algorithms

The temporal difference (TD) method is a combination of the MC method considered above and the bootstrapping DP method based on the Bellman equation. The main difference between the TD and MC methods is the target, the return G, in the incremental average in Eq. (20.49) for estimating either the state or action value functions. While in the MC method the target is the actual return G_{t} calculated at the end of the episode when all subsequent rewards r_{t+1}, ⋯, r_{T} are available, here in the TD method the target, called the TD target, is the sum of the immediate reward r_{t+1} and the previously estimated value v_{π}(s_{t+1}) at the next state:

G_{t} = r_{t+1} + γv_{π} (s_{t+1})  (20.51)

i.e., the TD method is an in-place bootstrapping method. It makes more efficient use of the sample data, and it updates the estimates of the value functions and policy more frequently at every step of an episode, instead of at the end of the episode as in the MC method. Also, it can be used even if the dynamic process of the environment is non-episodic with an infinite horizon.

Again, we first consider the simpler problem of policy evaluation. As in the MC method, we estimate the value function v_{π}(s) as the average of the actual returns G found by running multiple episodes while sampling the environment. Substituting this TD target into the running average in Eq. (20.49) for the value function above, we get

v_{π}(s) ⇐ v_{π}(s) + α(G − v_{π}(s))
= v_{π}(s) + α(r + γv_{π}(s′) − v_{π}(s)) = v_{π}(s) + αδ  (20.52)

where δ, called the TD error, is defined as the difference between the TD target and the previous estimate:

δ = G − v_{π}(s) = r + γv_{π}(s′) − v_{π}(s)  (20.53)

and v_{π}(s′) is the current estimate of the value of the next state previously estimated (bootstrapping). The iteration of this TD method converges to the true value v_{π}(s), if the step size is small enough.

Here is the pseudo code for policy evaluation using the TD method, based on parameters γ and α:

Input: a given policy π to be evaluated Initialize: v(s) for all s ∈ S arbitrarily, v(s) = 0 for terminal state s, α ∈ (0, 1] loop (for each episode)

s = s_{0}

while s is not terminal (for each step) take action a = π(s), get reward r and next state s′ v(s) = v(s) + α[r + γv(s′) − v(s)] s = s′

We further consider the TD method for model-free control, which is similar to the TD method for model-free evaluation considered before. By running many episodes of the environment, the estimated action values q(s, a) for all stateaction pairs, stored in a state-action table or Q-table, are iteratively updated as the running average of the return G, the sum of the immediate reward r and the action value q(s′, a′) of the next state based on an action a′, dictated by the policy being followed, such as the ϵ-greedy policy. This iterative process gradually learns the action value functions of the MDP model of the environment, based on which the optimal action for each state can be obtained as the one with the maximum Q-value.

The TD method can be carried out by two algorithms, with the only difference in terms of how the action value is updated.

• On-policy algorithm:

The action value q(s, a) is updated based on action a′ at the current state s dictated by the policy being followed:

q(s, a) ⇐ q(s, a) + α(G − q(s, a))
= q(s, a) + α(r + γq(s′, a′) − q(s, a))  (20.54)

• Off-policy algorithm:

The action value q(s, a) is updated based on the greedy action a′ at the current state s regard the policy being followed:

q(s, a) ⇐ q(s, a) + α(r + γ max_{a′} q(s′, a′) − q(s, a))  (20.55)

These two algorithms become the same if the policy being followed is greedy.

• State-Action-Reward-State-Action (SARSA)

This is an on-policy algorithm, which is called SARSA as it updates the Q-value based on the current state s and action a, the immediate reward r, the next state s′ and action a′. The pseudo code of SARSA algorithm is listed below, where the policy being followed, e.g., ϵ-greedy, is denote by π.

The pseudo code of SARSA algorithm is listed below. Initialize: q(s, a) = 0 for all (s, a), α ∈ (0, 1], ϵ > 0 loop (for each episode) s = s_{0} get action a at s according to π based on q(s, a) while s is not terminal (for each step)

take action a and get reward r and next state s′ get action a′ according to π q(s, a) = q(s, a) + α[r + γq(s′, a′) − q(s, a)] s = s′, a = a′

A variation of SARSA, the expected SARSA, updates the Q-value in state s based on the expected Q-value of the next state s′, the weighted average of the Q-values resulting from all possible actions, instead of only one action. Consequently, the estimated Q-values by the expected SARSA have lower variance than SARSA, and a higher learning rate α can be used to speed up the learning process.

q(s, a) = q(s, a) + α (r + γ ∑_{a} π(a|s′)q(s′, a) − q(s, a))  (20.56)

• Q-Learning

This is an off-policy algorithm, which updates the Q-value based on the greedy action instead of that dictated by the policy being followed.

The pseudo code of the Q-learning algorithm is listed below. Initialize: q(s, a) = 0 for all (s, a), α ∈ (0, 1], ϵ > 0 loop (for each episode)

s = s_{0}

while s is not terminal (for each step) take action a at s according to π based on q(s, a) get reward r and next state s′ q(s, a) = q(s, a) + α[r + γ max_{a′} q(s′, a′) − Q(s, a)] s = s′

Here is the comparison of the MC and TD methods:

• The MC method estimates the state value v_{π}(s_{t}), the expected return, by the true return G_{t} obtained at the end of the episode, i.e., the estimated value is updated only once at the end of every episode. The TD method estimates v_{π}(s_{t}) as the sum of the immediate reward and the action value at the next state previously estimated, i.e., it is a bootstrap method, and the estimated value is updated at every step of every episode.

• MC can only learn episodic (terminating) environments with complete episodes; while TD can learn continuing (non-terminating) environments of incomplete episodes.

• MC estimates based on sample returns G_{t}, and it is unbiased especially in first-visit version. The TD uses the bootstrap approach to find the TD target based on sample data that are not i.i.d. in general, and it is more sensitive to the initial guess of the value functions, so it is more biased.

• MC based on the sample returns G_{t} is affected by many random events (state transitions, actions, and rewards), and in particular the first-visit version only makes use of the sample data from the first visit of a state. It does not use the available data efficiently, and it has high variance. The TD method is based on only one random variable, the estimated return, and it updates the action value at every step of the episode, it makes more frequent and efficient use of the sample data, it has lower variance.

Figure 20.7 Comparison of DP, MC and TD Methods

The dynamic programming (DP) method for model-based planning, and the Monte-Carlo (MC) and time difference (TD) methods for model-free control are summarized in Fig. 20.7.

Example 20.1 The TD method is applied to the grid world problems as shown in Fig. 20.8, where a rectangular array is used to represent a maze of which all positions are sequentially indexed from top left to lower right. This problem is modeled as a Markov decision process with each of the positions treated as a state of the MDP model. For an agent to achieve the goal of finding the shortest path from the starting position (top left corner) to the destination position (lower right corner), we assign a large positive reward (e.g., 10) to the destination position, a large negative reward (e.g., -100) to all positions for the walls of the maze, and a small negative reward (e.g., -1) to all other empty space available for the agent to move into. By maximizing the cumulative reward along its way, the agent will find the desired path.

This maze problem can be solved by the following steps:

• Set up all parameters in the algorithm, including ϵ, α, γ, and the maximum number of iterations for episodes (outer loop) and steps in each of each episode (inner loop).

• Define the maze as a rectangular array with reward for each position (wall or empty space), together with the start and end positions (as the initial and goal states).

• Initialize the Q-table, here an array of the same size as the maze, to zero.

• Carry out either SARSA or Q-learning algorithm to learn the state-action functions by sampling the environment and updating the Q-table iteratively.

• Get the solution, the optimal path from the initial state to the goal state, with maximum cumulative reward, by taking the greedy action with the maximum Q-value in the Q-table for each state along the path.

The Matlab code listed below are the key functions used for solving the grid world problem.

function Q=SARSA(Q,s0,st)           % SARSA Learning
global maze alpha gamma epsilon max_step max_episode
ie=1;                           % episode counter
while ie < max_episode          % for all episodes
is=1;                       % episode counter
s=s0;                       % initial state
a=get_action(s,Q);          % action by e-greedy
while s~=st                 % for each step in episode
sp=get_next_state(s,a); % next state s'
r=get_reward(s,a);      % reward
ap=get_action(sp,Q);    % naxt action by e-greedy
Q(s,a)=Q(s,a)+alpha*(r+gamma*Q(sp,ap)-Q(s,a));
is=is+1;
s=sp;                   % update state
a=ap;                   % update action
if is>max_step
    break;
end
end
ie=ie+1;
end
end
function Q=Q_Learning(Q,s0,st)      % Q-Learning
global maze alpha gamma epsilon max_step max_episode
ie=1;                           % episode counter
while ie<max_episode            % for all episodes
is=1;                       % step counter
s=s0;                       % initial state
while s~=st                 % for each step in episode
a=get_action(s,Q);      % action by e-greedy policy
sp=get_next_state(s,a); % next state s'
r=get_reward(s,a);      % reward
Q(s,a)=Q(s,a)+alpha*(r+gamma*qmax(Q(sp,:))-Q(s,a));
s=sp;                   % update state

        is=is+1;
        if is>max_step
            break;
        end
    end
    ie=ie+1;
end
end
function r=get_reward(s,a)
                % get reward given state and action
global maze alpha gamma
sp = get_next_state(s,a);
[row,col] = state2idx(sp,maze);
r = maze(row,col);
end
function sp = get_next_state(s,a)
% get next state given current state s and action a.
global maze alpha gamma
[~,n]=size(maze);
switch a
    case 1       % move up
        sp=s-n;
    case 2       % move right
        sp=s+1;
    case 3       % move down
        sp=s+n;
    case 4       % move left
        sp=s-1;
end
end
function action = get_action(s, Q)
global maze alpha gamma epsilon
% get an action in state s by e-greedy policy
[M N]=size(maze);
[row,col]=state2idx(s,maze);
valid_actions=[];       % two-column array: action and Q-value
                        % check all four directions
if row > 1              % Up
    r=row-1;
    c=col;
    if maze(r,c)~=-100  % if valid
        valid_actions=[valid_actions; 1, Q(s,1)];

    end
end
if row < M              % Down
    r=row+1;
    c=col;
    if maze(r,c)~=-100  % if valid
        valid_actions=[valid_actions; 3, Q(s,3)];
    end
end
if col > 1              % Left
    r=row;
    c=col-1;
    if maze(r,c)~=-100  % if valid
        valid_actions=[valid_actions; 4, Q(s,4)];
    end
end
if col < N              % Right
    r=row;
    c=col+1;
    if maze(r,c)~=-100  % if valid
        valid_actions=[valid_actions; 2, Q(s,2)];
    end
end
if rand<epsilon         % of probability epsilon
    idx=randi(size(valid_actions,1)); % random index (non-greedy)
else                    % of probability 1-epsilon
    [q idx]=max(valid_actions(:,2));  % index of maximum Q (greedy)
end
action=valid_actions(idx,1);  % e-greedy action
end
function [r, c] = state2idx(s,maze)
% Convert state index s to row and column indices (r c)
[~,n]=size(maze);
r=ceil(s/n);
c=s-n*(r-1);
end
function [path reward]=find_path(s0, st, Q)
% get a path from s0 to st by taking greedy actions based on Q-table
global maze alpha gamma epsilon
Q(Q==0)=-inf;              % convert 0 (invalid actions) to -inf
reward=0;                  % initialize cumulative reward
path=[s0];                 % initialize path to start position

Figure 20.8 Optimal paths in four different mazes

    s=s0;
    while path(end)~=st        % add new positions to path
        [v,a]=max(Q(s,:));     % get greedy action a
        reward=reward+get_reward(s,a);   % update cumulative reward
        s=get_next_state(s,a); % get next position
        path=[path,s];         % append new position to path
    end
end

The optimal paths found by the code are illustrated in Fig. 20.8 for four slightly different mazes.

20.3.3 TD(λ) Algorithm

The MC and TD methods considered previously can be unified by the n-step TD(λ) method that spans a spectrum, of which the MC and TD methods are two special cases at the opposite extremes.

Recall that both MC and TD algorithms updates iteratively the estimated state-action value function q(s, a) based on Eq. (20.49), but they estimate the target G_{t} in the equation differently. In an MC algorithm, the target is the actual return G_{t} = r_{t+1} + γr_{t+2} + ⋯ + γ^{T −t−1}r_{T}, the sum of discounted rewards of all future steps up to the terminal state s_{T}, available only at the end of each episode. On the other hand, in a TD algorithm, the target is G_{t} = r_{t+1} + γv_{π}(s)t + 1) the sum of the immediate reward r_{t+1} available at each step of the episode, and the discounted value of the next state, based on bootstrapping. We therefore see that an MC algorithm updates the value functions once every episode, while a TD algorithm updates once every step in the episode.

As a tradeoff between the MC and TD methods, the n-step TD algorithm can be considered a generalization of the TD method, where the target in Eq. (20.49) is an n-step return, the sum of the discounted rewards in the n subsequent states and the discounted value function at the following state s_{t+n} with 1 ≤ n < ∞:

G_{t:t+n} = r_{t+1} + γr_{t+2} + ⋯ + γ^{n−1}r_{t+n} + γ^{n}v_{π}(s_{t+n})  (20.57)

In particular, when

• n = 1: the 1-step return is the sum of the immediate reward and the estimated value of the next state, the same as the TD target in the TD method:

G_{t:t+1} = r_{t+1} + γv(s_{t+1})  (20.58)

• n = T − t: the n-step return is the sum of the discounted rewards from all future states up to the terminal state at the end of the episode, i.e., it is the return r_{t} defined in Eq. (20.7), the same as in the MC method:

G_{t:t+n} = r_{t+1} + γr_{t+2} + γ^{2}r_{t+3} + ⋯ + γ^{T −t−1}r_{T} + v_{π}(s_{t+n}) = G_{t} (20.59)

where the value function of the terminal state v_{π}(s_{t+n}) = v_{π}(s_{T}) = 0 is zero.

• T − t < n < ∞: All states s_{n} beyond the terminal state s_{T} with n > T − t remain the same as the terminal state with value v(s_{T}) = 0 and return G_{t}, same as in the MC method.

We see that all these n-step returns form a spectrum with the TD (n = 1) and MC (n ≥ T − t) methods at the two ends.

Based on these n-step returns of different n values, the n-step TD algorithm can be further generalized to a more computationally advantageous and therefore more useful algorithm called TD(λ), by which the MC and TD algorithms are again unified as two special cases at the opposite ends of a spectrum.

We first define the λ-return G_{t}^{λ} as the weighted average of all n-step returns for n = 1, 2, ⋯, ∞:

G_{t}^{λ} = (1 − λ) ∑_{n=1}^{∞} λ^{n−1}G_{t:t+n}  (20.60)

where 0 ≤ λ < 1. Here the coefficient 1 − λ is needed for the exponentially decaying weights to be normalized:

(1 − λ) ∑_{n=1}^{∞} λ^{n−1} = (1 − λ) ∑_{n=0}^{∞} λ^{n} = \frac{1 − λ}{1 − λ} = 1  (20.61)

The λ-return G_{t}^{λ} defined above can be expressed in two summations: the sum of the first T −t−1 n-step returns G_{t:t+1}, ⋯, G_{t:T −1}, and the sum of all subsequent n-step returns G_{t:T} = ⋯ = G_{t:∞} = G_{t}:

G_{t}^{λ} = (1 − λ) [ ∑_{n=1}^{T −t−1} λ^{n−1}G_{t:t+n} + ∑_{n=T −t}^{∞} λ^{n−1}G_{t}]

= (1 − λ) ∑_{n=1}^{T −t−1} λ^{n−1}G_{t:t+n} + λ^{T −t−1}G_{t},  (0 ≤ t ≤ T) (20.62)

where the coefficient of G_{t} in the second term is the sum of all coefficients in the second summation:

(1 − λ) ∑_{n=T −t}^{∞} λ^{n−1} = (1 − λ) ∑_{m=0}^{∞} λ^{m}λ^{T −t−1} = λ^{T −t−1}  (20.63)

where m = n − T + t. We see that in Eq. (20.62) all n-step returns G_{t:t+n} are weighted, by exponentially decaying coefficient λ^{n−1}, except the true return G_{t} which is weighted by λ^{T −t−1}.

Again consider two special cases:

• λ = 0: all terms in Eq. (20.62) are zero, except the first one with n = 1 (with λ^{0} = 0^{0} = lim_{λ→0} λ^{0} = 1):

G_{t}^{λ=0} = λ^{n−1}G_{t:t+n}|_{n=1} = 0^{0} G_{t:t+1} = r_{t+1} + γv(s_{t+1})  (20.64)

This is simply the TD target, i.e., TD(0) is the TD method.

• λ = 1: the first summation is zero and

G_{t}^{λ=1} = G_{t}  (20.65)

i.e., TD(1) is the MC method.

We therefore see that TD(λ) is a general algorithm of which the two special cases TD(0) and TD(1) are respectively the TD and MC methods.

In Eq. (20.62), G_{t}^{λ} is calculated as the weighted sum of all n-step returns G_{t:t+n} up to the last one G_{t} available at the end of the episode. This summation can be truncated to include fewer terms before reaching the terminal state at the end of the episode:

G_{t:h}^{λ} = (1 − λ) ∑_{n=1}^{h−t−1} λ^{n−1}G_{t:t+n} + λ^{h−t−1}G_{t:h},  (0 ≤ t < h ≤ T) (20.66)

Specially when h = T, G_{t:h}^{λ} = G_{t}^{λ} is the same as in Eq. (20.62), and when h = t + 1, G_{t:h}^{λ} = G_{t:t+1} = r_{t+1} + γv(s_{t+1}) is the TD target.

Based on λ-return, the iterative update of the value function v_{π}(s_{t}) in Eq. (20.52) can be modified to:

v_{π}(s_{t}) = v_{π}(s_{t}) + α(G_{t}^{λ} − v_{π}(s_{t})) = v_{π}(s_{t}) + αδ_{t}  (20.67)

Figure 20.9 Forward View and Backward View

where

δ_{t} = G_{t}^{λ} − v_{π}(s_{t})  (20.68)

This iteration is called the forward view of TD(λ), which is similar to the MC method, as they are based on either G_{t} or G_{t}^{λ}, available in the future only at the end of each episode, as illustrated in Fig. 20.9.

An alternative but equivalent method is the backward view of TD(λ). This method is based the eligibility trace e_{t}(s) of each state, representing the frequency and recency of the visits to each state. If a state s has been more frequently and recently visited compared to others, its eligibility trace e_{t}(s) is greater than others and its value function v_{π}(s) will be updated by a greater increment than others.

The eligibility trace e_{t}(s) at all states are initialized to zero. At each step if state s = s_{t} is currently visited then its eligibility trace is boosted by an increment of 1 to become e_{t}(s) = e_{t}(s) + 1, while all others (if greater than zero) decay exponentially:

e_{t}(s) = γλe_{t−1}(s) (∀s ∈ S)  (20.69)

The backward view of TD(λ) is similar to the TD(0) method, in that they are both based on the immediate reward r_{t+1} available at each time step of the episode, instead of at the end of the episode, and they are therefore more computationally convenient. Here the iterative update of the estimated value function in Eq. (20.52) is modified so that all states (not only the one currently visited) are updated to different extents depending on e_{t}(s):

v_{π}(s) = v_{π}(s) + αδ_{t}e_{t}(s),  ∀s ∈ S  (20.70)

where the TD error δ_{t} is the same as in TD(0), given in Eq. (20.53):

δ_{t} = r_{t+1} + γv_{π}(s_{t+1}) − v_{π}(s_{t})  (20.71)

Here is the pseudo code for the backward view of the TD(λ) method for policy evaluation:

Input: policy π to be evaluated Initialize: v(s) = 0 ∀s ∈ S loop (for each episode) s = s_{0} e(s) = 0, ∀s ∈ S while s is not terminal (for each step)

take action a based on π(a|s), get reward r and next state s′ δ = r + γv(s′) − v(s) e(s) = e(s) + 1 for all s

v(s) = v(s) + αδe(s)
e(s) = γλe(s)

s = s′

Note that values at all states are updated, but to different extents depending on e(s), different from the TD algorithm where only the value at the state currently visited is updated.

Again consider two special cases:

• λ = 0, e(s) = 0 for all states except the current s being visited, i.e., TD(λ) becomes TD(0), the same as the TD method.

• λ = 1, e(s) is scaled down by a factor γ < 1 for all states except the current s being visited, i.e., TD(λ) becomes TD(1), the same as the MC method. However, different from the MC method that updates the value function v(s) at the end of each episode, here v(s) is still updated at every step of the episode due to the backward view of the method.

This backward view of the TD(λ) method can be applied to model-free control when the state value function is replaced by the action value function. Here are two algorithms based on eligibility traces that correspond to the SARSA and Q-learning algorithms based on TD(0).

• SARSA(λ) algorithm:

Initialize: q(s, a) = 0, ∀(s, a), α ∈ (0, 1], ϵ > 0, denote ϵ-greedy policy

π(a|s) by π

loop (for each episode) s = s_{0} e(s, a) = 0, ∀(s, a) get action a according to π based on q(s, a) while s is not terminal (for each step) e(s, a) = e(s, a) + 1 take action a, get reward r and next state s′ get action a′ according to π based on q(s′, a) δ = r + γq(s′, a′) − q(s, a) for all (s, a)

q(s, a) = q(s, a) + αδ e(s, a)
e(s, a) = γλe(s, a)

s = s′, a = a′

• Q(λ) algorithm:

Initialize: q(s, a) = 0, ∀(s, a), α ∈ (0, 1], ϵ > 0, denote ϵ-greedy policy

pi(a|s) by π

loop (for each episode) s = s_{0} e(s, a) = 0, ∀(s, a) get action a according to π based on q(s, a) while s is not terminal (for each step) e(s, a) = e(s, a) + 1 take action a, get reward r and next state s′ get action a′ according to π based on q(s′, a) a^{∗} = arg max_{b} q(s′, b) δ = r + γq(s′, a^{∗}) − q(s, a) for all (s, a)

q(s, a) = q(s, a) + αδ e(s, a)

if a′ = a^{∗} then e(s, a) = γλe(s, a) else e(s, a) = 0 s = s′, a = a′

Q(λ) algorithm is different from the SARSA(λ) algorithm in two ways. First, the action value is updated based on the greedy action a^{∗}, instead of the a′ give by policy π; second, if the ϵ-greedy policy π happens to choose a random non-greedy action a′ ≠ a^{∗} with probability ϵ to explore rather than exploiting a^{∗}, the eligibility traces of all states are reset to zero. There are other different versions of the algorithm which do not reset these traces.

20.4 Value Function Approximation

All previously considered algorithms are based on either the state value function v_{π}(s) for each state s or the action value function q_{π}(s, a) for each state-action pair. As these function values can be considered as either a 1-D or 2-D table, the algorithms based on such value functions are called tabular methods. However, when the number of states and the number of actions in each state are large (e.g., the game Go has 10^{170} states), the tabular methods are not suitable due to the unrealistic table size. In such a case, we can approximate the state value function v(s) or action-value function q(s, a) by a function v̂(s, θ) or q̂(s, a, θ), parameterized by a vector θ, containing a set of parameters of the function as its components. Such a function can be treated as a black box as illustrated in Fig. 20.10. Now the state or action value functions can be approximated based on θ to be estimated during the learning process when sampling the environment. This approach is more efficient and practical than the tabular methods due to its much smaller number of unknown parameters in comparison to the number of unknown variables in the tabular methods.

Such value function approximators can be implemented by methods such as those listed below:

Figure 20.10 Estimation of State and Action Value Functions

• Regression

• Multi-layer neural networks

• Decision trees

• Fourier/wavelet bases

We consider the approach of regression method in this section, and then the neural network method in the next section.

The regression method is to fit a continuous function v̂(s, θ) or q̂(s, a, θ) to a set of discrete and finite data points v_{π}(s) or q_{π}(s, a) obtained by sampling the environment, so that all points in the state space, including those for states or state-action pairs never actually observed during the sampling process, can be estimated by interpolation based on the regression function.

Our task is to find the optimal parameter θ of the regression model which minimizes the objective function, the mean squared error between the approximated values and the sample values:

ε(θ) = \frac{1}{2}E_{π}[(v_{π}(s) − v̂(s, θ))^{2}]  (20.72)

The gradient vector of ε(θ) is

g_{ε}(θ) = \frac{d}{dθ} ε(θ) = \frac{1}{2}E_{π} [\frac{d}{dθ} [(v_{π}(s) − v̂(s, θ))^{2}]]
= −E_{π} [(v_{π}(s) − v̂(s, θ))\frac{d}{dθ} v̂(s, θ)]  (20.73)

and the optimal parameter vector can be found iteratively by the gradient descent method:

θ_{n+1} = θ_{n}+Δθ = θ_{n}−αg_{ε}(θ) = θ_{n}+αE_{π} [(v_{π}(s) − v̂(s, θ))\frac{d}{dθ} v̂(s, θ)] (20.74)

where α is the step size or learning rate, and

Δw = −αg_{ε}(w) = αE_{π} [(v_{π}(s) − v̂(s, w))\frac{d}{dw} v̂(s, w)]  (20.75)

If the method of stochastic gradient descent is used (SGD, Section6.1), i.e., the parameter θ is updated based on only one data point at each iteration instead of the expectation of the data points, then the expectation symbol E_{π} can be dropped.

As the true value function v_{π}(s), the expected return, is unknown, it needs to be estimated by either the actual return G_{t} at each state s = s_{t} based on the MC method, or r_{t+1} + γv̂_{π}(s′, θ) in terms of the immediate reward r_{t+1} and the estimated value v̂_{π}(s′, θ) of the next state s′ based on the TD method. Such an approach can be considered as a supervised learning based on labeled training data set: {(s_{t}, G_{t}), ∀t} for the MC method, or {(s_{t}, r_{t+1}, ∀t} for the TD method.

Here we consider the special case where the state value v_{π}(s^{n}) of each of the N = |S| states is approximated by a linear regression function:

v̂(s^{n}, w) = ∑_{i=1}^{d} x_{i}(s^{n})w_{i} = w^{T} x(s^{n}) = w^{T} x_{n},  n = 1, ⋯, N  (20.76)

Here the generic symbol θ as the parameter of the regression is replaced by w, representing specifically the weights of the linear function. This model is based on the assumption that the value function of each state s^{n} can be indeed approximated as a linear combination of a set of features as the components in the d-dimensional feature vector x_{n} = x(s^{n}) = [x_{1}(s^{n}), ⋯, x_{d}(s^{n})]^{T}, which are hand picked or designed to properly represent the specific problem at hand.

As a special example, we can represent each of the N = |S| states by a N dimensional binary feature vector x_{n} = x(s^{n}) = [x_{1}(s^{n}), ⋯, x_{N}(s^{n})]^{T}, where x_{i}(s^{n}) is 1 if i = n but 0 otherwise. Then the weight vector becomes w = [w_{1}, ⋯, w_{N}]^{T} = [v_{π}(s^{1}), ⋯, v_{π}(s^{n})]^{T}, and we have v̂_{π}(s^{n}, w) = w^{T} x_{n} = v_{π}(s^{n}). In this case, each state s is represented by its value function v_{π}(s), the same as in all previous discussions.

The objective function for the mean squared error of such a linear approximating function is

ε(w) = \frac{1}{2}E_{π}[(v_{π}(s) − v̂(s, w))^{2}] = \frac{1}{2}E_{π}[(v_{π}(s) − w^{T} x(s))^{2}]  (20.77)

If all data points in terms of state s represented by x(s) and the corresponding return G(s) as a sample of the true value v_{π}(s) have already been collected, then the value function approximation can be carried out as an off-line algorithm in a batch manner, the same as in the linear least squares regression discussed in Chapter 5, and the optimal weight vector w^{∗} that minimizes the squared error

w^{∗} = arg min_{w} ∑_{n=1}^{N}[G(s^{n}) − w^{T} x_{n}]^{2}  (20.78)

can be found by solving a linear equation system X^{T} w = g, where X = [x_{1}, ⋯, x_{N}] is a d × N matrix containing all N states each represented by x_{n}, and vector g = [G(s^{1}), ⋯, G(s^{n})]^{T} contains the corresponding returns. As we assume the number of states is greater than the number of the model parameters, i.e., N ≥ d, this is an over-constrained system with the least squares so1ution (Eq. (5.10)):

w^{∗} = (XX^{T})^{−1}Xg = (X^{T})^{−}g  (20.79)

However, as reinforcement learning is typically an online process, the parameter w for the approximator needs to be modified in real-time whenever a new piece of data becomes available while sampling the environment. Then we can use the gradient descent method, based on the gradient vector of ε(w):

g_{ε}(w) = \frac{d}{dw} ε(w) = −E_{π}(v_{π}(s) − v̂(s, w))\frac{d}{dw} v̂(s, w)
= −E_{π}(v_{π}(s) − w^{T} x(s))x(s)  (20.80)

and find the optimal parameter w^{∗} iteratively:

w_{n+1} = w_{n} + Δw = w_{n} + αE_{π} [v_{π}(s) − v̂(s, w)] \frac{d}{dw} v̂(s, w)
= w_{n} + αE_{π} [v_{π}(s) − w^{T} x(s)] x(s)  (20.81)

with increment

Δw = αE_{π} [v_{π}(s) − v̂(s, w)] \frac{d}{dw} v̂(s, w) = αE_{π} [v_{π}(s) − w_{n}^{T} x(s)] x(s) (20.82)

If stochastic gradient descent (SGD) is used, then w is updated whenever a new data point x(s_{t}) is available at a new step t in the sampling process, and the expectation symbol E_{π} can be dropped. The expectation of the estimated w by this SGD method is the same as that obtained by full GD method. This iteration will convergence to the global minimum of the objective function ε(w) in Eq. (20.77), as it is a quadratic function with only one minimum which is global.

As the state value function v_{π}(s) is unknown, it needs to be estimated in one of several different ways, similar to the corresponding algorithms discussed before.

• MC method

The value function v_{π}(s) is estimated by the actual return G_{t} for each state s_{t} visited in an episode, obtained only at the end of each episode while sampling the environment:

Δw = α[G_{t} − v̂(s_{t}, w)]\frac{d}{dw} v̂(s_{t}, w) = α[G_{t} − w^{T} x(s_{t})]x(s_{t}) (20.83)

Here is the pseudo code for the algorithm based on the MC method, similar to that for value evaluation algorithms discussed previously: Initialize w = 0 loop (for each episode)

run current episode to the end to get G_{t}, t = 1, ⋯, T for t = 1, ⋯, T

if s_{t} is visited the first time

w = w + α[G_{t} − w^{T} x(s_{t})]x(s_{t}) end if

end for end loop • TD(0) method

The value function v_{π}(s) is estimated by the TD target, the sum of the immediate reward r_{t+1} and the approximated value of the next state s_{t+1}, at each time step of each episode while sampling the environment:

Δw = α[r_{t+1} + γw^{T} x(s_{t+1}) − w^{T} x(s_{t})]x(s_{t})  (20.84)

Here is the pseudo code for the algorithm based on the TD method, similar to that for value evaluation algorithms discussed previously: Initialize w = 0 loop (for each episode)

for each time step in current episode take action a = π(s), get reward r and next state s′ w = w + α[r + γw^{T} x(s′) − w^{T} x(s)]x(s)

end for end loop

• TD(λ) method

The value function v_{π}(s) is estimated based on the λ-return G_{t}^{λ} given in Eq. (20.62). As discussed before, TD(λ) has two flavors:

– Forward view: same as MC method, except the true return G_{t} is replaced by λ-return G_{t}^{λ} available only at the end of each episode:

Δw = α[G_{t}^{λ} − w^{T} x(s_{t})]x(s_{t})  (20.85)

– Backward view: similar to TD(0), this is based on the immediate reward r_{t+1} available at every step of each episode:

δ_{t} = α[r_{t+1} + γw^{T} x(s_{t+1}) − w^{T} x(s_{t})]x(s_{t})  (20.86)

Here the eligibility trace becomes a vector, which is initialized to zero at the beginning of the episode, and then updated by the gradient of the estimated value function v̂(s, w) = w^{T} x(s) (the components of e are updated differently instead all by 1):

e_{t} = γλe_{t−1} + \frac{d}{dw} w^{T} x(s_{t}) = γλe_{t−1} + x(s_{t})  (20.87)

while the weight vector is still updated the same as before:

Δw = αδ_{t}e_{t}  (20.88)

Given a policy π for an MDP, we also define a probability distribution d(s) over all states visited according to π, satisfying

∑_{s} d(s) = 1  (20.89)

and the following balance equation:

d(s′) = ∑_{s} ∑_{a} π(a|s)p(s′|s, a)d(s)  (20.90)

Figure 20.11 Policy Iteration Based on Approximated Value Functions

As both d(s′) and d(s) represent the same distribution, they must be identical.

Given the probability distribution d(s), the mean squared error of the value function approximation for a policy π can be expressed as

MSE(w) = ∑_{s} d(s)[v_{π}(s) − v̂_{π}(s, w)]^{2}  (20.91)

It can be shown that the MC method for the linear value function approximation will converge to the optimal weight vector w_{MC} that minimizes the mean squared error above:

MSE(w_{MC}) = min_{w} ∑_{s} d(s)[v_{π}(s) − v̂_{π}(s, w)]^{2}  (20.92)

20.5 Control Based on Function Approximation

Model-free control can be carried out based on the state action value function q̂(s, a, w) approximated as discussed in the previous section, by the approach of general policy iteration, as illustrated in Fig. 20.11. This is similar to the algorithms for model-free control illustrated in Fig. 20.5, with the only difference that the action-value function q_{π}(s, a) is now replaced by the parameter w of the linear model of the action value function:

q̂(s, a, w) = ∑_{n} w_{n}x(s, a) = w^{T} x(s, a)  (20.93)

where x(s, a) is the feature vector for the state-action pair (s, a). We need to find the optimal parameter w that minimizes the objective function, here specifically the mean square error of the approximation:

ε(w) = \frac{1}{2}E_{π}[(q_{π}(s, a) − q̂(s, a, w))^{2}] = \frac{1}{2}E_{π}[(q_{π}(s, a) − w^{T} x(s, a))^{2}] (20.94)

with gradient vector:

g_{ε}(w) = \frac{d}{dw} ε(w) = −E_{π} [q_{π}(s, a) − q̂(s, a, w)] \frac{d}{dw} q̂(s, a, w)
= −E_{π} [(q_{π}(s, a) − w^{T} x(s, a))x(s, a)]  (20.95)

If the stochastic gradient descent method is used based on a single sample of the action value q_{π}(s, a), instead of its expectation, then the expectation symbol E_{π} can be dropped. The optimal weight vector w^{∗} that minimizes ε(w) in Eq. (20.94) can be learned iteratively:

w_{t+1} = w_{t} + Δw = w_{t} − αg_{ε}(w)
= w_{t} + α [(q_{π}(s_{t}, a_{t}) − q̂_{π}(s_{t}, a_{t}, w))\frac{d}{dw} q̂_{π}(s_{t}, a_{t}, w)]
= w_{t} + α [(q_{π}(s_{t}, a_{t}) − w_{t}^{T} x(s_{t}, a_{t}))x(s_{t}, a_{t})]  (20.96)

where Δw is the increment of the update:

Δw = −αg_{ε}(w) = α[q_{π}(s_{t}, a_{t}) − q̂_{π}(s_{t}, a_{t}, w)]\frac{d}{dw} q̂_{π}(s_{t}, a_{t}, w)
= α[q_{π}(s_{t}, a_{t}) − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})  (20.97)

As the true Q-value q_{π}(s, a) in the expression is unknown, it needs to be estimated by some target depending on the specific methods used:

• MC method:

The true q_{π}(s, a) is replaced by the sample return G_{t} as the target, obtained at the end of each episode:

Δw = α[G_{t} − q̂(s_{t}, a_{t}, w)]\frac{d}{dw} q̂(s_{t}, a_{t}, w) = α[G_{t} − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})

(20.98)

• TD(0) method:

The action value function q_{π}(s, a) is replaced by the TD target, the sum of the immediate reward, available at each step of each episode, and the approximated action value of the next state s_{t+1}:

– SARSA (on-policy):

Δw = α[r_{t+1} + γq̂(s_{t+1}, a_{t+1}, w) − q̂(s_{t}, a_{t}, w)]\frac{d}{dw} q̂(s_{t}, a_{t}, w)
= α[r_{t+1} + γw^{T} x(s_{t+1}, a_{t+1}) − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t}) (20.99)

Following Eq. (20.71), the TD error is defined as:

δ_{t} = r_{t+1} + γq̂(s_{t+1}, a_{t+1}, w) − q̂(s_{t}, a_{t}, w)  (20.100)

then we get

Δw = αδ_{t} \frac{d}{dw} q̂(s_{t}, a_{t}, w)  (20.101)

– Q-learning (off-policy):

Δw = α[r_{t+1} + γ max_{a′} q̂(s_{t+1}, a′, w) − q̂(s_{t}, a_{t}, w)]\frac{d}{dw} q̂(s_{t}, a_{t}, w)
= α[r_{t+1} + γ max_{a′} w^{T} x(s_{t+1}, a′) − w_{t}^{T} x(s_{t}, a_{t})]x(s, a) (20.102)

• TD(λ) method:

In the forward-view version of the TD(λ) method, the action function q_{π}(s, a) is approximated by λ-return G_{t}^{λ} as the target, available only at the end of each episode:

Δw = α[G_{t}^{λ} − w_{t}^{T} x(s_{t}, a_{t})]x(s_{t}, a_{t})  (20.103)

In the backward-view version of the TD(λ) method, the eligibility trace in Eq. (20.87) becomes:

e_{t} = γλe_{t−1} + \frac{d}{dw} q̂(s_{t}, a_{t}, w)  (20.104)

and the weight vector is updated as in Eq. (20.88):

Δw = αδ_{t}e_{t}  (20.105)

where δ_{t} is the TD error for the backward view of TD(λ) first given in Eq. (20.71), but now with the state value v_{π}(s)) replaced by the estimated action value q̂(s, a, w):

δ_{t} = (r_{t+1} + γq̂_{π}(s_{t+1}, a_{t+1}) − q̂_{π}(s_{t}, a_{t}))  (20.106)

In summary, here are the conceptual (not necessarily algorithmic) steps for the general model-free control based on approximated action-value function:

• Learn parameter w as in Eq. (20.96),

• Get the Q-values as in Eq. (20.93)

• Obtain the policy by ϵ-greedy approach as in Eq. (20.42)

These steps are also illustrated below:

Training of w ⟹ Q-value ⟹ Policy π  (20.107)

20.6 Deep Q-learning

In the previous section, the Q-values at a given state are approximated by a linear regression function q̂(s, a, w) (Eq. (20.93).) parameterized by the weight vector w. However, when applied to a real world problem with large numbers of states and actions, this simple linear function may not be able to model the potentially highly complex nonlinear relationship between the state-action pairs and the value functions. In this case, a more sophisticated nonlinear model may be needed, such as a deep learning networks considered in Section 18.4 used for modeling complex nonlinear relationship y = f(x, W) between its input x and output y. When applied to reinforcement learning, such a deep learning network takes a given state s as input, and generates the approximated Q-values q̂(s, a, W) for all |A| actions available at the state as output. Here W represents all weights in the deep network, while vector w contains all weights of a linear regression function q̂(s, a, w). This deep learning network in reinforcement learning is called the deep Q-learning network (DQN), while all previously considered methods such as the methods of tabular Q-learning and linear function approximation are considered shallow. The three methods are illustrated in Fig. 20.12.

As we know, the method of tabular Q-learning models the environment by the Q-table of size |A||S| for all actions at all states, which may become impractically large when both |S| and |A| are large. This difficulty is avoided in DQN method, as well as in regression function approximation, which estimates the action value functions q̂(s, a) without exhausting all |A||S| cases. The cases not ever seen during training are approximated by interpolation based on those that have been seen. Also, the DQN method is superior to function approximation due to its multi-layer nonlinear learning capability. Moreover, while both methods take the current state s as input, function approximation is based on a small set of carefully handcrafted states as input, but the input to the DQN can be in some raw data with potentially high dimensionality extracted directly from the environment. For example, in a board game problem, it is very difficult to come up with a set of well defined states that characterizes all possible situations in the game. But for the DQN, the current state can be represented simply by the image of the board, of which all pixel values are directly fed into the input layer of the DQN (similar to the image input to an CNN considered in Section 18.4).

As a supervised learning method, the DQN is trained in the same way as a back-propagation network. The weights in W as the model parameter are iteratively updated by gradient descent, to gradually reduce the error, the difference between the actual output q̂(s, a, W) of the DQN and the desired target output, obtained by the Bellman equation (Eq. (20.33)) based on data collected while sampling the environment, including the current state s, the action a, the next state s′, and the reward r, collectively treated as an experience tuple e = (s, a, s′, r):

q(s, W′) = r + γ max_{a′} q(s′, a′, W′)  (20.108)

Here the second term is the highest output value from the DQN based on the previous estimated weights, denoted by W′. We denote the target q(s, W′) simply as q as it is treated as a constant while updating W. Now the squared error between the DQN output and the target can be found is:

ε(W) = \frac{1}{2} [q − q̂(s, a, W)]^{2} = \frac{1}{2} [r + γ max_{a′} q(s′, a′, W′) − q̂(s, a, W)]^{2}

(20.109)

Figure 20.12 Three Q-Learning Methods: Tabular, Function Approximation, and Deep Q-Learning

and its gradient

g_{ε}(W) = \frac{d}{dW} ε(W) = − [q(s, W′) − q̂(s, a, W)] \frac{d}{dW} q̂(s, a, W)
= − [r + γ max_{a′} q(s′, a′, W′) − q̂(s, a, W)] \frac{d}{dW} q̂(s, a, W)(20.110)

The weights in W are updated by gradient descent:

W_{n+1} = W_{n} − αg_{ε}(W)
= W_{n} + α [r + γ max_{a′} q(s′, a′, W′) − q̂(s, a, W)] \frac{d}{dW} q̂(s, a, W)(20.111)

For this deep Q-learning method there are two issues that need to be further addressed.

• First, to update W by Eq. (20.111) we need to estimate the target by the Bellman equation in Eq. (20.108). In this bootstrapping process, the resulting target q(s, W′) depends on the previous estimate q(s′, a′, W′), which in turn depends on the previous W′, i.e., the iterative training process is actually chasing a moving target. This is different from the training of a conventional back-propagation network, where the desired output, the target, is fixed. As the consequence, the training iteration of the DQN may be unstable, even diverging.

This problem is addressed by including a second network, called the target network and parameterized by W′. The output q′(s′, a′, W′) of this target network is used as the targets for training the first network, of which the output q(s, a, W) is used for the state transition while sampling the environment. To avoid chasing a moving target, here W′ of the target network remain unchanged for some period of C steps, while W of the first network is updated. At the end of the C-step period, the updated W is copied over to the target network to replace its W′.

• Second, during the sequential sampling of the environment step by step, the Q values obtained for the current state depends on those collected in the previous state, i.e., they are not i.i.d. samples as they are correlated. Consequently the training process may be biased. This issue is similar to that considered in Subsection 20.3.1, where we only use the data collected during the first visit, instead of every visit, to a state while sampling the environment.

This problem is addressed by a method called experience replay based on a buffer D = {e_{1}, ⋯, e_{N}} containing the agent’s previous experience tuples collected in the latest N steps while sampling the environment. The DQN is then trained based on a minibatch of such experiences randomly selected from the buffer D. As these experiences are in random order and no longer correlated, the training is unbiased.

Once the DQN has been trained, the agent can form the optimal policy by taking the action a corresponding to the maximum q(s, a) out of all outputs of the DQN for each state s as the input.

Here is the pseudo code for the deep Q-learning method:

Input: the states and their corresponding rewards in the environment (e.g., images of game board and the scores)

Output: Q-values for all actions in states Initialize replay buffer D for N experiences Initialize W = W′ for both the target network and DQN loop (for each episode)

for each time step in current episode Take action a at current state s based on ϵ-greedy policy:

a = \{ \begin{matrix}random action & P rob = ϵ \\ arg max_{a} q(s, a, W) & P rob = 1 − ϵ \end{matrix}

Get reward r and next state s′

Store a (s, a, s′, r) in D Move to next state:s = s′ For each experience (s, a, s′, , r) in a random minibatch from

D Find target: :

q = \{ \begin{matrix}r & if s′ is a terminal state \\ r + γ max_{a′} q′(s′, a′, W′) \end{matrix}

Update W by gradient descent based on gradient g_{ϵ}(W) of

squared error ε(W) = (q − q(, s, a, W))^{2} Update weights: W′ = W (i.e., q′ = q) every C steps

end for end loop

20.7 Policy Gradient Methods

All RL algorithms previously considered are based on either state or action value function and the policy is indirectly derived from them by greedy or ϵ-greedy method. However, as the ultimate goal of an RL problem is to find the optimal policy that maximizes the return, it can also be considered as an optimization problem to directly find the optimal policy that maximizes an objective function representing the total return, which can be addressed by the gradient ascent method, as discussed below.

In previous methods we approximated the state value function v_{π}(s) or action value function q_{π}(s, a) by a function v̂_{π}(s, w) or q̂_{π}(s, a, w), of which the parameter w is obtained by sampling the environment, as a supervised learning process. Now we will directly construct a model of the policy as a function parameterized by θ:

π_{θ}(a|s) = P (a|s, θ)  (20.112)

where θ = [θ_{1}, ⋯, θ_{d}]^{T} is a vector composed of d parameters of the policy model. Once θ is available, the probability for any action at any state is determined. This parameterized policy model is suitable in cases where the numbers of states and actions are large or even continuous.

Figure 20.13 Value-Based, Policy-Based, and Actor-Critic

As a specific example, the policy model can be based on the soft-max function:

π_{θ}(a|s) = P (a|s, θ) = \frac{e^{h(s,a,θ)}}{∑_{a′} e^{h(s,a′,θ)}}  (20.113)

satisfying ∑_{a} π_{θ}(a|s) = 1. Here the summation is over all possible actions in state s, and h(s, a, θ) is the preference of action a in state s, which can be a parameterized function such as a simple linear function h(s, a, θ) = θ^{T} x(s, a), or a neural network with weights as the parameters in θ, the same as how the value functions are approximated in Section 20.4. According to this policy, an action with higher preference h(s, a, θ) will have a higher probability to be chosen at state s.

Different from how we find the parameter w of q̂(s, a, w) by minimizing the objective function ε(w) in Eq. (20.94) for the mean squared error between the value function q_{π}(s, a) and its model q̂_{π}(s, a, w), here we find the parameter θ of π_{θ}(a|s) by maximizing an objective function J(θ) representing the value function, the expected return, under the policy, which is to be minimized by gradient ascent. Such a method is therefore called a policy gradient method.

The value-function based methods considered previously and the policy-based methods considered here are summarized below, together with theactor-critic method, as the combination of the two. These methods are also illustrated in Fig. 20.13

• Value-based: the policy is derived by the greedy or ϵ-greedy method based on the value function learned by sampling the environment as a supervised training process.

• Policy-based: the policy is directly learned without explicitly value function estimation.

• Actor-Critic: parameters for both value function model and policy model are learned simultaneously.

A policy-based method can learn stochastic policies effectively in high-dimensional or continuous action space with better convergence properties, but it may get stuck at a local optimum, and has high variance.

How good a policy is may be measured by different objective functions related to the values or rewards associated with the policy being evaluated, depending on the environment of the specific problem:

• In episodic environments with finite horizon, we can use the value, the expected return of the start state s_{0}:

J_{1}(θ) = v_{π_{θ}}(s_{0})  (20.114)

• In continuing (online) environments with infinite horizon, we can use the average value

J_{avv}(θ) = ∑_{s} d_{π_{θ}}(s)v_{π_{θ}}(s)  (20.115)

where d_{π_{θ}}(s) is the stationary distribution of all states under policy π(a|s, θ).

We can find the optimal parameter θ^{∗} for the policy model π_{θ}(a|s) by solving the maximization problem:

θ^{∗} = arg max_{θ} J(θ)  (20.116)

based on the gradient of J(θ):

▽_{θ}J(θ) = \frac{d}{dθ} J(θ) = [\frac{∂J}{∂θ_{1}} , ⋯, \frac{∂J}{∂θ_{n}} ]^{T}  (20.117)

by the iterative gradient ascent method:

θ_{t+1} = θ_{t} + Δθ = θ_{t} + α ▽_{θ} J(θ)  (20.118)

Here we use t as the index of the iteration based on the assumption that a new sample point is available at every time step of an episode while sampling the environment following policy π_{θ}(a|s).

While it is conceptually straight forward to see how the optimal parameter θ^{∗} that maximizes J(θ) can be found by the gradient ascent method, it is not easy to actually obtain the gradient ▽_{θ}J(θ), which depends on not only what actions to take at the states directly determined by the policy π_{θ}(a|s), but also how the states are distributed under the policy in an unknown environment. This challenge is addressed by the following policy gradient theorem, stating that the gradient of J(θ) can be calculated as shown below without explicit derivatives of the state distribution:

▽_{θ}J(θ) = E_{π_{θ}} [▽_{θ} ln π_{θ}(a|s) q_{π_{θ}}(s, a)]  (20.119)

Proof of policy gradient theorem:

Consider the derivative of the value function v_{π_{θ}} = ∑_{a} π_{θ}(a|s) q_{π_{θ}}(s, a) (in Eq. (20.21)):

▽_{θ}v_{π_{θ}}(s) = ▽_{θ} (∑_{a} π_{θ}(a|s)q_{π_{θ}}(s, a))
= ∑_{a} [▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a) + π_{θ}(a|s) ▽_{θ} q_{π_{θ}}(s, a)]

=^{1} ∑_{a} [▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a) + π_{θ}(a|s) ▽_{θ} (r(s, a) + γ ∑_{s′} P (s′|s, a)v_{π_{θ}}(s′))]

=^{2} ∑_{a} [▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a) + π_{θ}(a|s)γ ∑_{s′} P (s′|s, a) ▽_{θ} v_{π_{θ}}(s′))]
= ∑_{a} ▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a) + ∑_{a} π_{θ}(a|s)γ ∑_{s′} P (s′|s, a) ▽_{θ} v_{π_{θ}}(s′)
=^{3} ϕ(s) + ∑_{a} π_{θ}(a|s)γ ∑_{s′} P (s′|s, a) ▽_{θ} v_{π_{θ}}(s′)  (20.120)

where the labeled equal signs are due to the following:

1. Eq. (20.22)

2. r(s, a) is not a function of θ

3. We have defined

ϕ(s) = ∑_{a} ▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a)  (20.121)

Eq. (20.120) is a recursion by which ▽_{θ}v_{π_{θ}}(s) is expressed as a function in terms ▽_{θ}v_{π_{θ}}(s′).

We further consider the probability of transitioning from state s to a state x after k steps following π(a|s):

s ⟶^{π(a|s)} s′ ⟶^{π(a|s′)} s′′ ⟶^{π(a|s′′)} ⋯ ⟶^{π(a|s^{k−1})} s^{k} = x  (20.122)

denoted by P_{π}(s → x, k) with the following properties:

•

P_{π}(s → s, 0) = 1  (20.123)

•

P_{π}(s → s′, 1) = ∑_{a} π(a|s)P (s′|s, a)  (20.124)

•

∑_{s′} P_{π}(s → s′, 1) = 1  (20.125)

•

P_{π}(s → s′′, 2) = P_{π}(s → s′, 1)P_{π}(s′ → s′′, 1)  (20.126)

•

P_{π}(s → x, k + 1) = ∑_{s′} P_{π}(s → s′, k)P_{π}(s′ → x, 1)  (20.127)

Continuing Eq. (20.120) we keep rolling out the recursion of ▽_{θ}v_{π_{θ}}(s′) and get:

▽_{θ}v_{π_{θ}}(s) = ϕ(s) + γ ∑_{a} π_{θ}(a|s) ∑_{s′} P (s′|s, a) ▽_{θ} v_{π_{θ}}(s′)
=^{1} ϕ(s) + γ ∑_{s′} P_{π_{θ}}(s → s′, 1) ▽_{θ} v_{π_{θ}}(s′)
=^{2} ϕ(s) + γ ∑_{s′} P_{π_{θ}}(s → s′, 1) [ϕ(s′) + γ ∑_{s′′} P_{π_{θ}}(s′ → s′′, 1) ▽_{θ} v_{π_{θ}}(s′′)]
=^{3} ϕ(s) + γ ∑_{s′} P_{π_{θ}}(s → s′, 1)ϕ(s′) + γ^{2} ∑_{s′′} ∑_{s′} P_{π_{θ}}(s → s′, 1)P_{π_{θ}}(s′ → s′′, 1) ▽_{θ} v_{π_{θ}}(s′′)
=^{4} ϕ(s) + γ ∑_{s′} P_{π_{θ}}(s → s′, 1)ϕ(s′) + γ^{2} ∑_{s′′} P_{π_{θ}}(s → s′′, 2) ▽_{θ} v_{π_{θ}}(s′′)
=^{5} ϕ(s) + γ ∑_{s′} P_{π_{θ}}(s → s′, 1)ϕ(s′) + γ^{2} ∑_{s′′} P_{π_{θ}}(s → s′′, 2)ϕ(s′′) + γ^{3} ∑_{s′′′} P_{π_{θ}}(s → s′′′, 2) ▽_{θ} v_{π_{θ}}(s′′′)
= · · · · · · =^{6} ∑_{x∈S} ∑_{k=0}^{∞} γ^{k}P_{π_{θ}}(s → x, k)ϕ(x)  (20.128)

where the labeled equal signs are due to the following:

1. Eq. (20.124)

2. unroll ▽_{θ}v_{π_{θ}}(s′)

3. Eq. (20.125)

4. Eq. (20.127)

5. unroll ▽_{θ}v_{π_{θ}}(s′′)

6. unrolling recursively to infinity; Eq. (20.123)

Now the gradient of the objective J(θ) = v_{π_{θ}}(s_{0}) can be written as

▽_{θ}J(θ) = ▽_{θ}v_{π}(s_{0}) = ∑_{s∈S} ∑_{k=0}^{∞} γ^{k}P_{π_{θ}}(s_{0} → s, k) ϕ(s)
= ∑_{s∈S} η(s) ϕ(s) = (∑_{s′} η(s′)) ∑_{s∈S} \frac{η(s)}{∑_{s′} η(s′)} ϕ(s)
∝ ∑_{s∈S} \frac{η(s)}{∑_{s′} η(s′)} ϕ(s) = ∑_{s∈S} μ_{π}(s) ∑_{a} ▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a)(20.129)

where we also defined

η(s) = ∑_{k=0}^{∞} γ^{k}P_{π_{θ}}(s_{0} → s, k),  (20.130)

as the sum of all probabilities for visiting state s from the start state s_{0}, and

μ_{π}(s) = \frac{η(s)}{∑_{s′} η(s′)}  (20.131)

as a normalized version of η(s) representing the probability distribution of visiting state s while following policy π. Here the proportionality is due to the dropping of the constant ∑_{s′} η(s′) independent of state s.

Following the similar steps, the policy gradient theorem for environment with continuous state and action spaces can be also proven:

▽_{θ}J(θ) = ∫_{s} μ_{π}(s) ∫_{a} ▽_{θ}π_{θ}(a|s) q_{π_{θ}}(s, a)da ds  (20.132)

Now Eq. (20.129) can be further written as

▽_{θ}J(θ) = ▽_{θ}v_{π}(s_{0}) ∝ ∑_{s∈S} μ_{π}(s) ∑_{a} π_{θ}(a|s)\frac{▽_{θ}π_{θ}(a|s)}{π_{θ}(a|s)} q_{π_{θ}}(s, a)
= ∑_{s∈S} μ_{π}(s) ∑_{a} π_{θ}(a|s) ▽_{θ} ln π_{θ}(a|s) q_{π_{θ}}(s, a)
= E_{π_{θ}} [▽_{θ} ln π_{θ}(a|s) q_{π_{θ}}(s, a)]  (20.133)

where E_{π_{θ}} denotes the expectation over all actions in each state s weighted by π_{θ}(a|s) and all states s ∈ S weighted by μ_{π}(s).

Q.E.D.

The gradient ▽_{θ}J(θ) is now expressed as the expectation of the action value function q_{π}(s, a), weighted by the gradient of the logarithm of the corresponding policy π_{θ}(a|s). Now the gradient ascent iteration in Eq. (20.118) can be further written as

θ_{t+1} = θ_{t} + α ▽_{θ} J(θ) = θ_{t} + αE_{π_{θ}} [q_{π_{θ}}(s_{t}, a_{t}) ▽_{θ} ln π_{θ}(a_{t}|s_{t})] (20.134)

Now Eq. (20.118) can also be written as

θ_{t+1} = θ_{t}+α▽_{θ}J(θ) = θ_{t}+αE_{π_{θ}} [(q_{π_{θ}}(s_{t}, a_{t}) + b(s_{t})) ▽_{θ} ln π_{θ}(a_{t}|s_{t})] (20.135)

The expectation E_{π_{θ}} in Eqs. (20.134) and (20.135) can be dropped if the method of stochastic gradient ascent is used, as in all algorithms below, where each iterative step is based on only one sample data point instead of its expectation.

In particular, for a soft-max policy model as given in Eq. (20.113), we have

▽_{θ} ln π_{θ}(a|s) = ▽_{θ} ln (\frac{e^{θ^{T} x(s,a)}}{∑_{b} e^{θ^{T} x(s,b)}} ) = ▽_{θ} (θ^{T} x(s, a) − ln ∑_{b} e^{θ^{T} x(s,b)})

= x(s, a) − \frac{▽_{θ} ∑_{b} e^{θ^{T} x(s,b)}}{∑_{b} e^{θ^{T} x(s,b)}} = x(s, a) − \frac{∑_{b} e^{θ^{T} x(s,b)}x(s, b)}{∑_{b} e^{θ^{T} x(s,b)}}
= x(s, a) − ∑_{b} \frac{e^{θ^{T} x(s,b)}}{∑_{b} e^{θ^{T} x(s,b)}} x(s, b)
= x(s, a) − ∑_{b} π_{θ}(b|s)x(s, b)  (20.136)

A set of popular policy-based algorithms is listed below. They are based on either the MC or TD methods discussed before.

• REINFORCE (MC) policy gradient

In this algorithm, the action value function q_{π_{θ}}(s_{t}, a_{t}) in Eq. (20.134) is replaced by the return G_{t} obtained at the end of each episode while sampling the environment. Now the equation becomes:

θ_{t+1} = θ_{t} + αγ^{t}G_{t} ▽_{θ} ln π_{θ}(a|s)  (20.137)

Note that the discount factor γ^{t} is included as the expression for ▽_{θ}J(θ) in Eq. (20.133) assumed γ = 1 for simplicity.

Here is the pseudo code for the algorithm: Initialize π(a|s, θ loop (for each episode)

At the end of episode, get G_{t} for each state visited for t = 1, ⋯, T

θ = θ + αγ^{t}G_{t} ▽_{θ} ln π_{θ}(a_{t}|s_{t})

end loop

• Actor-Critic (TD) policy gradient

As its name suggests, this algorithm is based on two approximation function models, the first for the policy π_{θ}(a|s) parameterized by θ, the actor, the same as in the REINFORCE algorithm above, and the second for the value function v̂_{π}(s, w) parameterized by w, the critic, the same as in Eq. (20.96).

In Eq. (20.135), the action value function q_{π_{θ}}(s_{t}, a_{t}) is replaced by its bootstrapping expression r_{t+1} + γv̂_{π}(s′, w), and the bias term b(s) is replaced by the approximated value function v̂(s_{t}, w). Now both parameters w and θ can be found iteratively at every step of an episode while sampling the environment (the TD method) by stochastic gradient method with the expectation symbol E_{π} dropped:

w_{t+1} = w_{t} + α_{w} [(r_{t+1} + γv̂_{π}(s′, w) − v̂_{π}(s, w)) ▽_{w} v̂_{π}(s, w)]
θ_{t+1} = θ_{t} + α_{θ} [(r_{t+1} + γv̂_{π}(s′, w) − v̂_{π}(s, w)) ▽_{θ} ln π_{θ}(a|s)](20.138)

Here is the pseudo code for the algorithm: initialize model parameters θ and w initialize step sizes α_{θ} and α_{w} loop (for each episode)

initialize s = s_{0}, t = 0 while s is not terminal (for each step)

take action a following π_{θ}(a|s), find reward r and next

state s′ find TD error: δ = r + γv̂(s′, w) − v̂(s, w) w = w + α_{w} δ ▽ v̂(s, w) θ = θ + α_{θ}γ^{t}δ ▽ ln π_{θ}(a|s) s = s′ t = t + 1

end while end loop

• Backward view of TD(λ) policy gradient

Here is the pseudo code for the algorithm: initialize model parameters θ and w initialize step sizes α_{θ} and α_{w} initialize trace-decay rates λ_{θ} and λ_{w} loop (for each episode)

initialize s = s_{0}, t = 0 initialize z_{θ} = 0, z_{w} = 0 while s is not terminal (for each step)

take action a following π_{θ}(a|s), find reward r and next

state s′ find TD error: δ = r + γv̂(s′, w) − v̂(s, w) z_{θ} = γλ_{θ}z_{θ} + ▽γ^{t} ln π_{θ}(a|s) z_{w} = γλ_{w}z_{w} + γ^{t} ▽ v̂(s, w) w = w + α_{w} δ ▽ v̂(s, w) θ = θ + α_{θ} δ ▽ ln π_{θ}(a|s) s = s′ t = t + 1

end while end loop

Problems

The problems in this homework set are more involved than those in previous homework sets. They can be considered as a capstone final project for the entire course.

1. Solve the grid world problem in Example 20.1 by the TD method discussed in Section 20.3 in both flavors of SAESA and Q-learning.

2. Solve the same grid world problem by the MC method discussed in Section 20.3.

3. Solve the same grid world problem by the method of value function approximation discussed in Section 20.4 based on linear regression.

4. Solve the same grid world problem by the method of value function approximation discussed in Section 20.5, based on linear regression. Try both MC and TD methods.

5. Solve the same grid world problem by the method of value function approximation based on a nonlinear neural network instead of a linear regression model.

6. Solve the same grid world problem by the policy gradient method.

Appendix A A Review of Linear Algebra

A.1 Inner Product Space

A.1.1 Vector Space

• Vector space

A vector space over field F is a set V which is closed under the following two operations of addition and scalar multiplication defined for its members (called vectors), i.e., the results of the operations are also members of V. 1.The vector addition that maps any two vectors x, y ∈ V to another vector x + y ∈ V satisfying the following properties:

– Commutativity: x + y = y + x.

– Associativity: x + (y + z) = (x + y) + z.

– Existence of zero: there is a vector 0 ∈ V such that: 0+x = x+0 = x.

– Existence of inverse: for any vector x ∈ V, there is another vector −x ∈ V such that x + (−x) = 0.

2.The scalar multiplication that maps a vector x ∈ V and scalar a ∈ F (F can be a real or complex field) to another vector ax ∈ V with the following properties:

– a(x + y) = ax + ay.

– (a + b)x = ax + bx.

– abx = a(bx).

– 1x = x. where 0 = [0, ⋯, 0]^{T} and 1 = [1, ⋯, 1]^{T} are two constant vectors.

A subset W of V is a subspace of V, denoted by W ⊆ V, if it is also a vector space, i.e., it is closed under the same operations defined in V: 1.The zero vector 0 of V must be in W (the zero vector is unique in V, which W must have). 2.For any x, y ∈ W, x + y ∈ W. 3.For any x ∈ W, ax ∈ W.

Listed below is a set of typical vector spaces:

– n-D vector space ℝ^{n} or ℂ^{n} This space contains all n-D vectors expressed as an n-tuple, an ordered list of n elements (or components):

x = [ x_{1} ; x_{2} ; ⋮ ; x_{n} ] = [x_{1}, x_{2}, …, x_{n}]^{T} ,  (A.1)

which can be used to represent a discrete signal containing n samples. We will always represent a vector as a vertical or column vector, or the transpose of a horizontal or row vector. The space is denoted by either ℂ^{n} if the elements are complex x_{i} ∈ ℂ, or ℝ^{n} if they are all real x_{i} ∈ ℝ (i = 1, …, n). A subspace of ℝ^{n} can be a ℝ^{m} (m < n) that passes origin (zero). For example, any 2-D plane passing through the origin of a 3-D space is its subspace. However, if the 2-D plane does not pass through the origin, it is not a subspace. Also, as 3-D cube or sphere centered at the origin is not a subspace, as it is not closed under the operations of addition and scalar multiplication.

– A vector space can be defined to contain all m × n matrices composed of n m-D column vectors:

A = [a_{1}, …, a_{n}] = [ \begin{matrix}a_{11} & a_{12} & ⋯ & x_{1n} \\ x_{21} & x_{22} & ⋯ & x_{2n} \\ ⋮ & ⋮ & ... & ⋮ \\ x_{m1} & x_{m2} & ⋯ & x_{mn} \end{matrix} ] ,  (A.2)

where the ith column is an m-D vector a_{i} = [x_{1i}, …, x_{mi}]^{T}. Such a matrix can be converted to an mn-D vector by cascading all of the column (or row) vectors.

– l^{2} space:

The concept of an n-D space ℝ^{n} or ℂ^{n} can be generalized by allowing the dimension n to become to infinity so that a vector in the space becomes a sequence x = […, x_{i}, …]^{T} for 0 ≤ i < ∞ or −∞ < i < ∞. If all vectors are square-summable, the space is denoted by l^{2}. All discrete energy signals are vectors in l^{2}.

– L^{2} space:

A vector space can also be a set of real or complex valued continuous functions x(t) defined over either a finite range such as 0 ≤ t < T, or an infinite range −∞ < t < ∞. If all functions are square-integrable, the space is denoted by L^{2}. All continuous energy signals are vectors in L^{2}.

Note that the term “vector”, generally denoted by x in the following, may be interpreted in two different ways. First, in the most general sense, it represents a member of a vector space, such as any of the vector spaces considered above; e.g., a function x = x(t) ∈ L^{2}. Second, in a more narrow sense, it can also represent a tuple of n elements, an n-D vector x = [x_{1}, …, x_{n}]^{T} ∈ ℂ^{n}, where n may be infinity. It should be clear what a vector x represents from the context.

• Linear independence

A set of vectors {v_{1}, ⋯, v_{n}} are linearly independent if none of them can be represented as a linear combination of the others.

Theorem: For any set of linearly independent vectors {v_{1}, ⋯, v_{n}}, if their linear combination is zero

∑_{j=1}^{n} c_{j}v_{j} = 0  (A.3)

then all coefficients must be zero c_{1} = ⋯ = c_{n} = 0, or c = 0.

Proof: Assume c_{k} ≠ 0, then we would get

v_{k} = −\frac{1}{c_{k}} ∑_{j=1, i≠k}^{n} c_{j}v_{j}  (A.4)

i.e., v_{k} is a linear combination of the remaining n − 1 vectors, in contradiction with the assumption that they are independent.

In particular, in the n-D Euclidean space, the n vectors can be written as v_{j} = [v_{1j}, ⋯, v_{nj}]^{T} (j = 1, ⋯, n), and their linear combination

∑_{j=1}^{n} c_{j}v_{j} = [v_{1}, ⋯, v_{n}] [ c_{1} ; ⋮ ; c_{n} ] = [ \begin{matrix}c_{11} & ⋯ & c_{1n} \\ ⋮ & ... & ⋮ \\ c_{n1} & ⋯ & c_{nn} \end{matrix} ] [ c_{1} ; ⋮ ; c_{n} ] = Vc

(A.5)

For this to be a zero vector, i.e., for the homogeneous equation Vc = 0 to hold, the coefficient vector c has to be zero, as V is a full rank matrix. QED

• Convex combination

The convex combination of n vectors x_{1}, ⋯, x_{n} is their sum weighted by coefficients {c_{1}, ⋯, c_{n}} that add up to 1:

∑_{i=1}^{n} c_{i}x_{i},  ∑_{i=1}^{n} c_{i} = 1  (A.6)

The convex hull of these points is the set of all their convex combinations. For example, the convex hull of three points in a plane is the triangle formed by these points as vertices, in which any point is a convex combination of the three vertices.

• Inner product

An inner product in a vector space V is a function that maps two vectors x, y ∈ V to a scalar ⟨x, y⟩ ∈ ℂ or ℝ and satisfies the following conditions:

– Positive definiteness:

⟨x, x⟩ ≥ 0,  ⟨x, x⟩ = 0 iff x = 0.  (A.7)

– Conjugate symmetry:

⟨x, y⟩ = \overline{⟨y, x⟩}.  (A.8)

If the vector space is real, the inner product becomes symmetric:

⟨x, y⟩ = ⟨y, x⟩.  (A.9)

– Linearity in the first variable:

⟨ax + by, z⟩ = a⟨x, z⟩ + b⟨y, z⟩,  (A.10)

where a, b ∈ ℂ. The linearity does not apply to the second variable unless the coefficients are real a, b ∈ ℝ:

⟨x, ay + bz⟩ = \overline{⟨ay + bz, x⟩} = \overline{a⟨y, x⟩ + b⟨z, x⟩} = \overline{a}⟨x, y⟩ + \overline{b}⟨x, z⟩

≠ a⟨x, y⟩ + b⟨x, z⟩

In the special case when b = 0, we have

⟨ax, y⟩ = a⟨x, y⟩, ⟨x, ay⟩ = \overline{a}⟨x, y⟩.  (A.11)

More generally we have

⟨∑_{n} c_{n}x_{n}, y⟩ = ∑_{n} c_{n}⟨x_{n}, y⟩, ⟨x, ∑_{n} c_{n}y_{n}⟩ = ∑_{n} \overline{c}_{n}⟨x, y_{n}⟩.

(A.12)

An inner product space is a vector space with inner product defined. In particular, when the inner product is defined, ℂ^{n} is called a unitary space and ℝ^{n} is called a Euclidean space.

Some examples of the inner product are listed below:

– In an n-D vector space, the inner product, also called the dot product, of two vectors x = [x_{1}, …, x_{n}]^{T} and y = [y_{1}, …, y_{n}]^{T} is defined as

⟨x, y⟩ = x^{T} \overline{y} = y^{∗}x = [x_{1}, …, x_{n}] [ \overline{y}_{1} ; ⋮ ; \overline{y}_{n} ] = ∑_{i=1}^{n} x_{i}\overline{y}_{i}, (A.13)

where y^{∗} = \overline{y}^{T} is the conjugate transpose of y. If y = \overline{y} is real, then y^{∗} = y^{T}.

– In a space of 2-D matrices containing m × n elements, the inner product of two matrices A and B is defined as

⟨A, B⟩ = ∑_{i=1}^{m} ∑_{j=1}^{n} a_{ij}\overline{b}_{ij}.  (A.14)

When the column (or row) vectors of A and B are concatenated to form two mn-D vectors, their inner product takes the same form as that of two n-D vectors.

– In a function space, the inner product of two function vectors x = x(t) and y = y(t) is defined as

⟨x(t), y(t)⟩ = ∫_{a}^{b} x(t)\overline{y(t)} dt = \overline{∫_{a}^{b} y(t)\overline{x(t)}} dt = \overline{⟨y(t), x(t)⟩}. (A.15)

– The covariance of two random variables x and y can be considered as an inner product

⟨x, y⟩ = σ_{xy}^{2} = E[(x − μ_{x})\overline{(y − μ_{y})}] = E(x\overline{y}) − μ_{x}\overline{μ}_{y}  (A.16)

Specially when μ_{x} = μ_{x} = 0, we have

⟨x, y⟩ = E[x\overline{y}].  (A.17)

• Vector norm

In general, the norm ||x|| of a vector x ∈ V is a non-negative real scalar that measures its size or length. ||x|| = 0 if and only if x = 0. There exist different definitions for vector norm, as shown in Section A.4.1. A vector x is normalized (becomes a unit vector) if ||x|| = 1. Any given vector can be normalized when divided by its own norm: x/||x||.

The most widely used 2-norm of a vector is defined as

||x|| = \sqrt{⟨x, x⟩} = ⟨x, x⟩^{1/2},  (A.18)

The vector norm squared ||x||^{2} = ⟨x, x⟩ can be considered as the energy of the vector. In particular, in an n-D unitary space, the 2-norm of a vector x = [x_{1}, …, x_{n}]^{T} ∈ ℂ^{n} is:

||x|| = \sqrt{⟨x, x⟩} = \sqrt{x^{T} \overline{x}} = \sqrt{x^{∗}x} = (∑_{i=1}^{n} x_{i}\overline{x}_{i})^{1/2} = (∑_{i=1}^{n} |x_{i}|^{2})^{1/2} .

(A.19)

The total energy contained in this vector is its norm squared:

E = ||x||^{2} = ⟨x, x⟩ = ∑_{i=1}^{n} |x_{i}|^{2}.  (A.20)

Similarly, in a function space, the norm of a function vector x = x(t) is defined as

||x|| = (∫_{a}^{b} x(t)\overline{x(t)} dt)^{1/2} = (∫_{a}^{b} |x(t)|^{2} dt)^{1/2} ,  (A.21)

where the lower and upper integral limits a < b are two real numbers, which may be extended to all real values ℝ in the entire real axis −∞ < t < ∞.

This norm exists only if the integral converges to a finite value; i.e., x(t) is an energy signal containing finite energy:

∫_{−∞}^{∞} |x(t)|^{2} dt < ∞.  (A.22)

All such functions x(t) satisfying the above are square-integrable, and they form a function space denoted by L^{2}(ℝ).

• Cauchy-Schwarz inequality

The Cauchy-Schwarz inequality holds for any two vectors x and y in an inner product space V:

|⟨x, y⟩|^{2} ≤ ⟨x, x⟩⟨ y, y⟩; i.e., 0 ≤ |⟨x, y⟩| ≤ ||x|| ||y||  (A.23)

Proof: If either x or y is zero, ⟨x, y⟩ = 0, the theorem holds (an equality). Otherwise, we consider the following inner product:

⟨x − λy, x − λy⟩ = ||x||^{2} − \overline{λ}⟨x, y⟩ − λ⟨y, x⟩ + |λ|^{2}||y||^{2} ≥ 0, (A.24)

where λ ∈ ℂ is an arbitrary complex number, which can be assumed to be:

λ = \frac{⟨x, y⟩}{||y||^{2}}, then \overline{λ} = \frac{⟨y, x⟩}{||y||^{2}}, |λ|^{2} = \frac{|⟨x, y⟩|^{2}}{||y||^{4}} .  (A.25)

Substituting these into the previous equation we get

||x||^{2} − \frac{|⟨x, y⟩|^{2}}{||y||^{2}} ≥ 0; i.e., |⟨x, y⟩| ≤ ||x|| ||y||.  (A.26)

The equation holds only if x − λy = 0 or x = λy, i.e., the two vectors are linearly dependent.

• Distance between two vectors

The distance d(x, y) between two vectors x and y is a real constant that satisfies:

– d(x, y) = 0 iff x = y

– d(x, y) = d(y, x)

– d(x, y) ≤ d(x, z) + d(z, y) In an inner-product space in which the inner product ⟨x, y⟩ between any two vectors x and y is defined, the distance between the two points can be defined as the norm of the difference between the two points:

d(x, y) = ||x − y|| = \sqrt{⟨(x − y), (x − y)⟩}  (A.27)

The norm of a vector can now be seen as its distance to the origin 0 of the space: ||x|| = ||x − 0||.

A vector space V is a metric space if a distance (or metric) d(x, y) between any two vector (two points) x and y is defined.

• Angle between two vectors The angle between two vectors x and y is defined as

θ = cos^{−1} (\frac{⟨x, y⟩}{||x|| ||y||}) .  (A.28)

Now the inner product of x and y can also be written as

⟨x, y⟩ = ||x|| ||y|| cos θ ≤ ||x|| ||y||  (A.29)

This is the Cauchy-Schwarz inequality. In particular,

– If θ = 0, cos θ = 1, then x and y are colinear or linearly dependent, and the inner product is maximized:

⟨x, y⟩ = ||x|| ||y||  (A.30)

i.e., the Cauchy-Schwarz inequality becomes an equality.

– If 0 < θ < π/2, 0 < cos θ < 1, we get the Cauchy-Schwarz inequality:

⟨x, y⟩ < ||x|| ||y||  (A.31)

– If θ = π/2, cos θ = 0, then x and y are orthogonal to each other, and the inner product is minimized:

⟨x, y⟩ = 0  (A.32)

Two vectors x and y are orthogonal or perpendicular to each other, denoted by x ⊥ y, if their inner product is zero ⟨x, y⟩ = 0, i.e., the angle between them is θ = cos^{−1} 0 = π/2.

• Projection

The orthogonal projection of a vector x ∈ V onto another vector y ∈ V is defined as a vector

p_{y}(x) = \frac{⟨x, y⟩}{||y||} \frac{y}{||y||} = \frac{⟨x, y⟩}{⟨y, y⟩}y = ||x|| cos θ \frac{y}{||y||},  (A.33)

where y/||y|| is the unit vector along the direction of y. In particular, if y is normalized with ||y|| = 1, then

p_{y}(x) = ⟨x, y⟩ y = ||x|| cos θ y  (A.34)

Note that

||p_{y}(x)|| = ||x|| cos θ  (A.35)

If only the magnitude of the projection is of interest, the unit vector y/||y|| can be dropped:

p_{y}(x) = ⟨x, y⟩/||y||  (A.36)

• Cauchy space

A sequence of points in a metric space x_{0}, x_{1}, x_{2}, ⋯ is a Cauchy sequence if it converges, i.e., for any ϵ > 0, there exists an integer N > 0 so that the following is true for any m, n > N:

d(x_{m}, x_{n}) < ϵ  (A.37)

If the limit of any Cauchy sequence of points in the space is also in the space, the space is complete, referred to as a Cauchy space.

A.1.2 Orthogonal Basis and Gram-Schmidt Process

Assume the n-D vector space ℝ^{n} or ℂ^{n} is spanned by a set of n independent basis vectors {v_{1}, ⋯, v_{n}}, not necessarily orthogonal, so that any vector x ∈ ℝ^{n} can be represented as a linear combination of the basis vectors:

x = ∑_{i=1}^{n} c_{i}v_{i} = [ v_{1} ⋯ v_{n} ] [ c_{1} ; ⋮ ; c_{n} ] = Vc  (A.38)

where V = [v_{1}, ⋯, v_{n}] is an n × n matrix composed of the n vectors and the coefficients in c = [c_{1}, ⋯, c_{n}]^{T} can be found by solving the linear equation system to get c = V^{−1}x with complexity is O(n^{3}).

These linearly independent vectors v_{1}, ⋯, v_{n} can be converted into a set of orthogonal vectors {u_{1}, ⋯, u_{n}} satisfying u_{i}^{T} u_{j} = 0 (i ≠ j) by the following Gram-Schmidt process:

u_{1} = v_{1}
u_{2} = v_{2} − p_{u_{1}}(v_{2}) = v_{2} − (\frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1}
u_{3} = v_{3} − p_{u_{1}}(v_{3}) − p_{u_{2}}(v_{3}) = v_{3} − (\frac{v_{3}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1} − (\frac{v_{3}^{T} u_{2}}{u_{2}^{T} u_{2}} ) u_{2}
· · · · · · · · ·
u_{k} = v_{k} − ∑_{i=1}^{k−1} p_{u_{i}}(v_{k}) = v_{k} − ∑_{i=1}^{k−1} (\frac{v_{k}^{T} u_{i}}{u_{i}^{T} u_{i}} ) u_{i}  (A.39)

where p_{u_{i}}(v_{k}) is the projection of v_{k} onto u_{i}:

p_{u_{i}}(v_{k}) = (\frac{v_{k}^{T} u_{i}}{u_{i}^{T} u_{i}} ) u_{i}  (A.40)

We see that u_{2} so obtained is indeed orthogonal to all u_{1}

u_{2}^{T} u_{1} = [v_{2} − (\frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} ) u_{1}]^{T} u_{1} = v_{2}^{T} u_{1} − \frac{v_{2}^{T} u_{1}}{u_{1}^{T} u_{1}} u_{1}^{T} u_{1} = 0  (A.41)

and, by mathematical induction, we can further prove that u_{k}^{T} u_{i} = 0 for all i = 1, ⋯, k − 1. In other words, u_{k} is the component of v_{k} that is orthogonal to all previously found orthogonal vectors u_{1}, ⋯, u_{k−1}.

Given a set of orthogonal basis {u_{1}, ⋯, u_{n}} satisfying ⟨u_{i}, u_{j}⟩ = u_{i}^{T} u_{j} = 0 for any i ≠ j, we can again represent any given vector x as a linear combination of these basis as

x = ∑_{i=1}^{n} d_{i}u_{i}  (A.42)

The coefficients d_{i} can now be obtained easily. Premultiplying u_{j}^{T} on both sides of the equation above, we get

u_{j}^{T} x = u_{j}^{T} (∑_{i=1}^{n} d_{i}u_{i}) = ∑_{i=1}^{n} d_{i}u_{j}^{T} u_{i} = d_{j}u_{j}^{T} u_{j}  (A.43)

Solving for c_{j} we get

d_{j} = \frac{u_{j}^{T} x}{u_{j}^{T} u_{j}} , j = 1, ⋯, n  (A.44)

As the n coefficients are decoupled, each of them can be obtained separately with linear computational complexity O(n) with total complexity O(n^{2}) for all n of them. The complexity is reduced to O(n^{2}) from O(n^{3}) for solving the equation system c = V^{−1}b, needed in the case where the basis is not orthogonal. This is the reason why orthogonal bases are preferred in general. Now the vector x can be written as

x = ∑_{i=1}^{n} d_{i}u_{i} = ∑_{i=1}^{n} (\frac{u_{j}^{T} x}{u_{j}^{T} u_{j}} ) u_{i} = ∑_{i=1}^{n} p_{u_{i}}(x)  (A.45)

where the ith term of the summation above is simply the projection p_{u_{i}}(x) of x onto the ith basis vector u_{i}.

The concept of orthogonal vectors satisfying u^{T} v = 0 can be generalized to conjugate vectors that satisfy u^{T} Av = 0 with respect to a symmetric matrix A = A^{T}. This can be expressed in the form of inner product

u^{T} Av = (Av)^{T} u = v^{T} Au  (A.46)

Conjugate vectors with respect to A can also be considered as orthogonal to each other with respect to A. Two orthogonal vectors satisfying u^{T} v = 0 can be considered as a special case of conjugate vectors with respect to A = I.

Based on this generalized orthogonality, we can also define the projection of v onto u with respect to A as:

p_{u}(v) = (\frac{u^{T} Av}{u^{T} Au}) u  (A.47)

Given a set of A-conjugate basis vectors {d_{1}, ⋯, d_{n}} satisfying d_{i}^{T} Ad_{j} = 0 for any i ≠ j, we can represent any x as

x = ∑_{i=1}^{n} a_{i}d_{i}  (A.48)

The coefficients a_{i} can be obtained by pre-multiplying d_{j}^{T} A on both sides:

d_{j}^{T} Ax = ∑_{i=1}^{n} a_{i}d_{j}^{T} Ad_{i} = a_{j}d_{j}^{T} Ad_{j}  (A.49)

Solving for a_{j} we get

a_{j} = \frac{d_{j}^{T} Ax}{d_{j}^{T} Ad_{j}} , j = 1, ⋯, n  (A.50)

and x is now represented as the sum of its A-projections onto the n basis vectors {d_{1}, ⋯, d_{n}}:

x = ∑_{i=1}^{n} a_{i}d_{i} = ∑_{i=1}^{n} (\frac{d_{i}^{T} Ax}{d_{i}^{T} Ad_{i}} ) d_{i} = ∑_{i=1}^{n} p_{d_{i}}(x)  (A.51)

A set of independent basis vectors {v_{1}, ⋯, v_{n}} can now also converted to an orthogonal basis with respect to a symmetric vector A by Gram-Schmidt process:

d_{k} = v_{k} − ∑_{i=1}^{k−1} p_{d_{i}}(v_{k}) = v_{k} − ∑_{i=1}^{k−1} (\frac{v_{k}^{T} Ad_{i}}{d_{i}^{T} Ad_{i}} ) d_{i},  (k = 1, ⋯, n) (A.52)

A.2 Matrices

A.2.1 Rank, Trace, Determinant, Transpose, and Inverse

Let A be an m × n square matrix:

A = [a_{1}, ⋯, a_{n}] = [ \begin{matrix}a_{11} & ⋯ & a_{1n} \\ ⋮ & ... & ⋮ \\ a_{m1} & ⋯ & a_{mn} \end{matrix} ]_{m×n}  (A.53)

where

a_{j} = [ a_{1j} ; ⋮ ; a_{mj} ] ,  (j = 1, ⋯, n)  (A.54)

is the jth column vector and [a_{i1} ⋯ a_{in}] is the ith row vector (i = 1, ⋯, m). If m = n, A is a square matrix. In particular, if all entries of a square matrix are zero except those along the diagonal, it is a diagonal matrix. Moreover, if the diagonal entries of a diagonal matrix are all one, it is the identity matrix:

I = [ \begin{matrix}1 & 0 & ⋯ & 0 \\ 0 & 1 & & ⋮ \\ ⋮ & & ... & 0 \\ 0 & ⋯ & 0 & 1 \end{matrix} ]  (A.55)

• Rank

The m row vectors span the row space of A and the n columns vectors span the column space of A. The rank of each space is its dimension, the number of independent vectors in the space. The row and column spaces have the same rank, which is also the rank of matrix A, i.e.:

r = rank(A) ≤ min(m, n)  (A.56)

In other words, the rank of matrix A is the number of its independent rows or columns.

• Transpose The transpose x^{T} a column vector x is a row vector:

x^{T} = [ x_{1} ; ⋮ ; x_{n} ]^{T} = [x_{1}, ⋯, x_{n}]  (A.57)

The transpose A^{T} of a matrix A is obtained by switching the positions of elements a_{ij} and a_{ji} for all i ∈ {1, ⋯, m} and j ∈ {1, ⋯, n}. In other words, the ith column of A becomes the ith row of A^{T}, or equivalently, the ith row of A becomes the ith column of A^{T}:

A^{T} = [a_{1} ⋯ a_{n}]^{T} = [ a_{1}^{T} ; ⋮ ; a_{n}^{T} ]  (A.58)

where vector a_{i} is the ith column of A and its transpose a_{i}^{T} is the ith row of A^{T}.

The properties of the transpose:

– (A^{T})^{T} = A

– (A + B)^{T} = A^{T} + B^{T}

– (AB)^{T} = B^{T} A^{T}

– A^{T} and A have the same eigenvalues and eigenvectors.

If A = A^{T}, it is a symmetric matrix.

• Conjugate transpose

The conjugate transpose of a matrix A, denoted by A^{∗}, is by taking the complex conjugate of the its transpose:

A^{∗} = (\overline{A})^{T} = \overline{A^{T}}  (A.59)

i.e., (A^{∗})_{ij} = \overline{A}_{ji}, the ij-th entry of A^{∗} is the complex conjugate of the ji-th entry of A.

The properties of the conjugate transpose:

– (A^{∗})^{∗} = A

– (cA)^{∗} = \overline{c}A^{∗}

– (A + B)^{∗} = A^{∗} + B^{∗}

– (AB)^{∗} = B^{∗}A^{∗}

– det(A^{∗}) = (det A)^{∗} = \overline{det A}

– tr(A^{∗}) = (trA)^{∗} = \overline{trA}

– (A^{∗})^{−1} = (A^{−1})^{∗}.

– The eigenvalues of A^{∗} are the complex conjugates of the eigenvalues of A. (But their eigenvectors are not related.)

– ⟨Ax, y⟩ = ⟨x, A^{∗}y⟩ Proof:

⟨Ax, y⟩ = (Ax)^{∗}y = x^{∗}A^{∗}y = x^{∗}(A^{∗}y) = ⟨x, A^{∗}y⟩  (A.60)

If A = A^{∗}, it is a Hermitian matrix. A real Hermitian matrix is a symmetric matrix. The inverse of an invertible Hermitian matrix is also Hermitian, i.e., if A^{∗} = A, then (A^{−1})^{∗} = A^{−1}.

• Trace The trace of a square matrix A is the sum of its diagonal elements:

tr(A) = ∑_{i=1}^{n} a_{ii}  (A.61)

The properties of the trace:

– tr(cA) = c tr(A)

– tr(A^{T}) = tr(A)

– tr(A + B) = tr(B + A)

– tr(AB) = tr(BA)

– tr(A^{T} B) = tr(AB^{T})

– tr(R^{−1}AR) = tr(R^{−1}(AR)) = tr((AR)R^{−1}) = tr(A)

• Determinant

The determinant of a square matrix A is denoted by det(A) = |A|, and det(A) ≠ 0 if and only if it is full rank, i.e., rank(A) = n.

The properties of the determinant:

– det(I) = 1

– det(A^{T}) = det(A)

– det(A^{−1}) = det(A)^{−1}

– det(AB) = det(A) det(B)

– det(cA) = c^{n} det(A)

• Inverse

If A is an m×n full rank square matrix with m = n = r, then there exists an inverse matrix A^{−1} that satisfies AA^{−1} = A^{−1}A = I. The inverse A^{−1} does not exist if A is not square (m ≠ n) or full rank (r < m = n).

The properties of the inverse:

– (A^{−1})^{−1} = A

– (A^{T})^{−1} = (A^{−1})^{T}

– det(A^{−1}) = det(A)^{−1}

– (AB)^{−1} = B^{−1}A^{−1} For example, the solution of a linear equation system Ax = b of m

equations and n = m variables can be obtained as x = A^{−1}b.

A.2.2 Normal, Unitary, and Similar Matrices

• Commutative matrices Two square matrices A and B commute if AB = BA. Obviously all diagonal matrices commute.

• Normal matrix

A square matrix A is normal if it commutes with its conjugate transpose: A^{∗}A = AA^{∗}. If A^{∗} = A^{T} is real, then A^{T} A = AA^{T}.

Obviously unitary matrices (A^{∗} = A^{−1}), Hermitian matrices (A^{∗} = A), and skew-Hermitian matrices (A^{∗} = −A) are all normal. But there exist normal matrices that are not Hermitian.

• Unitary matrix

U = [u_{1}, ⋯, u_{n}] is a unitary matrix if its conjugate transpose is equal to its inverse U^{∗} = U^{−1}, i.e., U^{∗} U = I. When a unitary matrix \overline{U} = U is real, it becomes an orthogonal matrix, U^{T} = U^{−1}.

The column (or row) vectors of a unitary matrix A are orthonormal, i.e. they are both orthogonal and normalized:

⟨u_{i}, u_{j}⟩ = u_{i}^{T} \overline{u}_{j} = u_{j}^{∗}u_{i} = ∑_{k} u_{ik} \overline{u}_{jk} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (A.62)

Let λ and v be an eigenvalue and the corresponding eigenvector of U, i.e., Uv = λv, then we have

v^{∗}v = v^{∗}U^{∗}Uv = (Uv)^{∗}Uv = λ^{2}v^{∗}v  (A.63)

We see that λ^{2} = 1, i.e., |λ| = 1.

A Hermitian matrix A can be converted to a diagonal matrix Λ (or diagonalized) by a particular unitary matrix U:

U^{∗}AU = Λ = diag[λ_{1}, ⋯, λ_{n}]  (A.64)

where Λ is a diagonal matrix, i.e., all its off diagonal elements are 0.

• Similar matrices

Two matrices A and B are similar if they are related by A = P^{−1}BP. Similar matrices have the same determinant, trace, and eigenvalues.

A.2.3 Positive/Negative (Semi)-Definite Matrices

Given a Hermitian matrix A = A^{∗} and any non-zero vector x ≠ 0, we can construct a quadratic form x^{∗}Ax. The matrix A is said to be

• positive definite A > 0, if x^{∗}Ax > 0

• positive semi-definite A ≥ 0, if x^{∗}Ax ≥ 0

• negative definite A < 0, if x^{∗}Ax < 0

• negative semi-definite A ≤ 0, if x^{∗}Ax ≤ 0

• indefinite if there exists x and y such that x^{T} Ax < 0 < y^{T} Ay.

For example, consider the covariance matrix of a random vector x

Σ_{x} = E[(x − m_{x})(x − m_{x})^{∗}]  (A.65)

The corresponding quadratic form is

v^{∗}Σ_{x}v = v^{∗}E[(x − m_{x})(x − m_{x})^{∗}]v
= E[v^{∗}(x − m_{x})(x − m_{x})^{∗}v] = E(s^{2}) ≥ 0

where s = v^{∗}(x − m_{x}) is a scalar. Therefore Σ_{x} is positive semi-definite.

As A is Hermitian, its eigenvalues λ_{1}, ⋯, λ_{n} are real and its eigenvector matrix is unitary V^{−1} = V^{∗}, by which it can be diagonalized:

V^{−1}AV = V^{∗}AV = Λ, i.e. A = VΛV^{∗}  (A.66)

where vector y = V^{∗}x is a unitary transform of vector x.

For any x = [x_{1}, ⋯, x_{n}]^{T}, we have

x^{∗}Ax = x^{∗}(VΛV^{∗})x = (V^{∗}x)^{∗}Λ(V^{∗}x) = y^{∗}Λy = ∑_{i=1}^{n} λ_{i}|y_{i}|^{2}

= \{ \begin{matrix}> 0 is positive definite & if λ_{i} > 0 \\ ≥ 0 is positive semi-definite & if λ_{i} ≥ 0 \\ < 0 is negative definite & if λ_{i} < 0 \\ ≤ 0 is negative semi-definite & if λ_{i} ≤ 0 \end{matrix} (for all i)

Also, if some eigenvalues are positive and some others are negative, x^{∗}Ax may be either positive or negative depending on x , i.e., A is indefinite.

As the eigenvalues of A^{−1} are 1/λ_{i}, i = (1, ⋯, n), A^{−1} and A share the same positive/negative definiteness.

A.2.4 Woodbury Matrix Identity and Sherman-Morrison Formula

The Woodbury matrix identity gives the inverse of an n × n square matrix A modified by a perturbation term UBV^{T}

(A + UBV^{T})^{−1} = A^{−1} − A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}  (A.67)

Proof:

(A + UBV^{T}) [A^{−1} − A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]

= [I − U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]
+ [UBV^{T} A^{−1} − UBV^{T} A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]
= I + UBV^{T} A^{−1} − [U(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1} + UBV^{T} A^{−1}U(B^{−1} + V^{T} A^{−1}U)^{−1}VA^{−1}]
= I + UBV^{T} A^{−1} − (U + UBV^{T} A^{−1}U)(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}]
= I + UBV^{T} A^{−1} − UB(B^{−1} + V^{T} A^{−1}U)(B^{−1} + V^{T} A^{−1}U)^{−1}V^{T} A^{−1}
= I + UBV^{T} A^{−1} − UBV^{T} A^{−1} = I  (A.68)

Consider some special cases:

• If U = V = I, then we get

(A + B)^{−1} = A^{−1} − A^{−1}(A^{−1} + B^{−1})^{−1}A^{−1}  (A.69)

• If B = I, and let U = [u_{1}, ⋯, u_{m}] and V = [v_{1}, ⋯, v_{m}], then we get the inverse of a rank-n modified matrix:

(A+UV^{T})^{−1} = (A + ∑_{i=1}^{m} u_{i}v_{i}^{T} )^{−1} = A^{−1}−A^{−1}U(I+V^{T} A^{−1}U)^{−1}V^{T} A^{−1}

(A.70)

• More specially, if m = 1, U = u and V = v become vectors, then we get the Sherman-Morrison formula:

(A + uv^{T})^{−1} = A^{−1} − \frac{A^{−1}uv^{T} A^{−1}}{1 + v^{T} A^{−1}u}  (A.71)

A.2.5 Inverse and Determinant of Partitioned Symmetric Matrices

• Theorem 1 (inverse of a partitioned symmetric matrix)

Divide an n × n symmetric matrix A into four blocks

A = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ]  (A.72)

The inverse matrix B = A^{−1} can also be divided into four blocks:

B = A^{−1} = [ \begin{matrix}B_{11} & B_{12} \\ B_{21} & B_{22} \end{matrix} ]  (A.73)

Here we assume the dimensionalities of these blocks are:

– A_{11} and B_{11} are p × p,

– A_{22} and B_{22} are q × q,

– A_{12} = A_{21}^{T} and B_{12} = B_{21}^{T} are p × q with p + q = n. Then we have

B_{11} = (A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1} = A_{11}^{−1} + A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}A_{21}A_{11}^{−1} B_{22} = (A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1} = A_{22}^{−1} + A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}A_{12}A_{22}^{−1} B_{21} = −A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1} B_{12} = −A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}

(A.74)

Proof:

I_{n} = AA^{−1} = AB = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ] [ \begin{matrix}B_{11} & B_{12} \\ B_{21} & B_{22} \end{matrix} ]
= [ \begin{matrix}A_{11}B_{11} + A_{12}B_{21} & A_{11}B_{12} + A_{12}B_{22} \\ A_{21}B_{11} + A_{22}B_{21} & A_{21}B_{12} + A_{22}B_{22} \end{matrix} ] = [ \begin{matrix}I_{p} & 0 \\ 0 & I_{q} \end{matrix} ]

544  A Review of Linear Algebra

Equate each of the four blocks to get:

A_{11}B_{11} + A_{12}B_{21} = I_{p}, or B_{11} = A_{11}^{−1} − A_{11}^{−1}A_{12}B_{21}

A_{11}B_{12} + A_{12}B_{22} = 0, or B_{12} = −A_{11}^{−1}A_{12}B_{22}
A_{21}B_{11} + A_{22}B_{21} = 0, or B_{21} = −A_{22}^{−1}A_{21}B_{11}
A_{21}B_{12} + A_{22}B_{22} = I_{q} or B_{22} = A_{22}^{−1} − A_{22}^{−1}A_{21}B_{12}

(A.75)

Plug B_{21} into B_{11} to get:

B_{11} = A_{11}^{−1} + A_{11}^{−1}A_{12}A_{22}^{−1}A_{21}B_{11}  (A.76)

Solve for B_{11} to get:

B_{11} = (A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}  (A.77)

Applying theorem 1 to this expression, we also get the other expression in the theorem. Similarly we can get

B_{22} = (A_{22} − A_{21}A_{11}^{−1}A_{12})^{−1}
B_{21} = −A_{22}^{−1}A_{21}(A_{11} − A_{12}A_{22}^{−1}A_{21})^{−1}
B_{12} = −A_{11}^{−1}A_{12}(A_{22} − A_{21}A_{11}^{−1}A_{12}^{−1})^{−1}

• Theorem 2 (Determinant of a partitioned symmetric matrix)

|A| = |[ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ]| = |A_{22}| |A_{11}−A_{12}A_{22}^{−1}A_{21}| = |A_{11}| |A_{22}−A_{21}A_{11}^{−1}A_{12}|

(A.78)

Proof:

A = [ \begin{matrix}A_{11} & A_{12} \\ A_{21} & A_{22} \end{matrix} ] = [ \begin{matrix}A_{11} & 0 \\ A_{21} & I \end{matrix} ] [ \begin{matrix}I & A_{11}^{−1}A_{12} \\ 0 & A_{22} − A_{21}A_{11}^{−1}A_{12} \end{matrix} ]
= [ \begin{matrix}I & A_{12} \\ 0 & A_{22} \end{matrix} ] [ \begin{matrix}A_{11} − A_{12}A_{22}^{−1}A_{21} & 0 \\ A_{22}^{−1}A_{21} & I \end{matrix} ]

The theorem is proved as we also know that

|BC| = |B| |C|  (A.79)

and

| \begin{matrix}B & 0 \\ X & C \end{matrix} | = | \begin{matrix}B & X \\ 0 & C \end{matrix} | = |B| |C|  (A.80)

A.2.6 Unitary Transform

Given any unitary matrix A satisfying AA^{∗} = A^{∗}A = I, we can define a unitary transform of a vector x = [x_{1}, ⋯, x_{n}]^{T}:

y = [ y_{1} ; ⋮ ; y_{n} ] = A^{∗}x = [  \overline{a}_{1}^{T} ; ⋮ ; \overline{a}_{n}^{T}  ] [ x_{1} ; ⋮ ; x_{n} ] , (forward transform)

x = [ x_{1} ; ⋮ ; x_{n} ] = Ay = [ a_{1} ⋯ a_{n} ] [ y_{1} ; ⋮ ; y_{n} ] = ∑_{i=1}^{n} y_{i} a_{i} (inverse transform)

(A.81)

When A = \overline{A} is real, A^{−1} = A^{T} is an orthogonal matrix and the corresponding transform is an orthogonal transform.

The first equation above is the forward transform and can be written in component form as:

y_{i} = \overline{a}_{i}^{T} x = ⟨x, a_{i}⟩ = ∑_{j=1}^{n} x_{j}\overline{a}_{ij}, (i = 1, ⋯, n)  (A.82)

The transform coefficient is an inner product y_{i} = ⟨x, a_{i}⟩, representing the projection of vector x onto the ith column vector a_{i} of the transform matrix A. The second equation is the inverse transform and can also be written in component form as:

x_{j} = ∑_{i=1}^{n} a_{ji} y_{i}, (j = 1, ⋯, n)  (A.83)

By this transform, vector x is represented as a linear combination (weighted sum) of the n column vectors a_{i}, a_{2}, ⋯, a_{n} of matrix A. Geometrically, x is a point in the n-dimensional space spanned by these n orthonormal basis vectors. Each coefficient (coordinate) y_{i} is the projection of x onto the corresponding basis vector a_{i}.

As the n-dimensional space can be spanned by the column vectors of any n×n unitary (orthogonal) matrix, a vector x in the space can be represented by any of such matrices, each defining a different transform.

Example A.1 When A = I = [⋯, e_{i}, ⋯] is an identity matrix, we have

x = ∑_{i=1}^{n} y_{i}a_{i} = ∑_{i=1}^{n} x_{i}e_{i}  (A.84)

where e_{i} = [0, ⋯, 0, 1, 0, ⋯, 0]^{T} is the ith column of I with the ith element equal 1 and all other 0.

Example A.2 When a_{m,n} = w[m, n] = e^{−j2πmn/N}, the corresponding transform is discrete Fourier transform. The nth column vector w_{n} of the transform matrix W = [w_{0}, ⋯, w_{N−1}] represents a sinusoid of a frequency nf_{0}, and the corresponding complex coordinate y_{n} = (x, w_{n}) represents the magnitude |y_{n}| and phase ∠y_{n} of this nth frequency component. The Fourier transform y = Wx represents a rotation of the coordinate system.

A unitary (orthogonal if real) transform y = A x can be interpreted geometrically as the rotation of the vector x about the origin, or equivalently, the representation of the same vector in a rotated coordinate system. A unitary transform y = A x does not change the inner product. Let u = A^{∗}x and v = A^{∗}y be the unitary transforms of vectors x and t, then the inner product of these two vectors is preserved:

⟨u, v⟩ = ⟨A^{∗}u, A^{∗}v⟩ = x^{∗}AA^{∗}y = x^{∗}y = ⟨x, y⟩  (A.85)

In particular, if x = y, we see that the vector norm is preserved by the unitary transform:

||u||^{2} = ⟨u, u⟩ = ⟨x, x⟩ = ||x||^{2}  (A.86)

This is the Parseval’s identity that indicates that the norm (length) of a vector is preserved under any unitary transform. If X is interpreted as a signal, then its length ||x||^{2} = ||y||^{2} represents the total energy or information contained in the signal, which is conserved during any unitary transform. However, some other features of the signal may change, e.g., the signal may be decorrelated and its total energy redistributed among its components after the transform, which may be desirable in many applications.

If x is a random vector with mean vector m_{x} and covariance matrix Σ_{x}:

m_{x} = E(x), Σ_{x} = E(xx^{∗}) − m_{x}m_{x}^{∗}  (A.87)

then its transform y = A^{∗}x has the following mean vector and covariance matrix:

m_{y} = E(y) = E(A^{∗}x) = A^{∗}E(x) = A^{∗}m_{x}  (A.88)

Σ_{y} = E(yy^{∗}) − m_{y}m_{y}^{∗} = E[(A^{∗}x)(A^{∗}x)^{∗}] − (A^{∗}m_{x})(A^{∗}m_{x})^{∗}
= E[A^{∗}(xx^{∗})A] − A^{∗}m_{x}m_{x}^{∗}A = A^{∗}[E(xx^{∗}) − m_{x}m_{x}^{∗}]A
= A^{∗}Σ_{x}A  (A.89)

In general the unitary transform of any square matrix A by a unitary matrix R is B = R^{∗}AR.

A.3 Eigenvalues and Eigenvectors

A.3.1 Eigenvalue Decomposition

For any n × n square matrix A, if there exist a vector v and a scalar λ such that the following eigenequation holds:

Av = λv,  (A.90)

then λ and v are called the eigenvalue and eigenvector of matrix A, respectively. In other words, the linear transformation of vector v by A has the same effect of scaling the vector by factor λ. (Note that for an m × n non-square matrix A with m ≠ n, Av is an m-D vector but λv is n-D vector, i.e., no eigenvalues and eigenvectors are defined.)

Given Av = λv, we also have Acv = λcv for any scalar constant c, i.e., the eigenvector v is not unique but up to any scaling factor. For the uniqueness of v, we typically keep it normalized so that ||v|| = 1.

To obtain λ, we rewrite the above equation as

(λI − A)v = 0  (A.91)

For this homogeneous equation system to have non-zero solutions for v, the determinant of its coefficient matrix has to be zero:

det(λI − A) = |λI − A| = 0  (A.92)

This is the characteristic polynomial equation of matrix A. Solving this nth order equation of λ, we get n eigenvalues {λ_{1}, ⋯, λ_{n}}. Substituting each λ_{i} back into the homogeneous equation system, we get the corresponding eigenvector v_{i}. We can put all n eigenequations Av_{i} = λ_{i}v_{i} together and obtain the more compact form:

AV = A[v_{1}, ⋯, v_{n}] = [λ_{1}v_{1}, ⋯, λ_{n}v_{n}]

= [v_{1}, ⋯, v_{n}] [ \begin{matrix}λ_{1} & 0 & ⋯ & 0 \\ 0 & λ_{2} & ⋯ & 0 \\ ⋮ & ⋮ & ... & ⋮ \\ 0 & 0 & ⋯ & λ_{n} \end{matrix} ] = VΛ  (A.93)

where we have defined

Λ = diag[λ_{1}, ⋯, λ_{n}] and V = [v_{1}, ⋯, v_{n}]  (A.94)

The eigenequation can be written in some alternative forms:

A = VΛV^{−1}, or V^{−1}AV = Λ  (A.95)

In the first form, A = VΛV^{−1} is expressed as a product of three matrices, called eigenvalue decomposition of the matrix; in the second form, A is diagonalized by its eigenvector matrix V to become a diagonal matrix, its eigenvalue matrix Λ.

Given Av = λv, we have the following:

• A^{T} has the same eigenvalues and eigenvectors as A.

Proof: As a matrix A and its transpose A^{T} have the same determinant, they have the same characteristic polynomial:

|A − λI| = |(A − λI)^{T} | = |A^{T} − λI|  (A.96)

therefore they have the same eigenvalues and eigenvectors.

• The eigenvalues and eigenvectors of A^{∗} are the complex conjugate of the eigenvalues and eigenvectors as A.

• A^{T} A has the same eigenvectors as A, but its eigenvalues are λ^{2}.

Proof:

A^{T} Av = A^{T} λv = λ^{2}v  (A.97)

• A^{k} has the same eigenvectors as A, but its eigenvalues are {λ_{1}^{k}, ⋯, λ_{n}^{k}},

where k is a positive integer.

Proof:

A^{2}v = AAv = Aλv = λ^{2}v  (A.98)

This result can be generalized to

A^{k}v = λ^{k}v  (A.99)

• In particular when k = −1, i.e., the eigenvalues of A^{−1} are {λ_{1}^{−1}, ⋯, λ_{n}^{−1}}. Proof: Left multiplying A^{−1} on both sides of Av_{i} = λv_{i} we get

A^{−1}Av_{i} = Iv_{i} = v_{i} = λ_{i}A^{−1}v_{i}  (A.100)

Dividing both sides by λ we get

A^{−1}v_{i} = \frac{1}{λ}_{i}v_{i} = λ_{i}^{−1}v_{i}  (A.101)

• The eigenvalues of a matrix are invariant under any unitary transform B =

R^{∗}AR, where R is unitary, satisfying R^{−1} = R^{∗}, or R^{∗}R = RR^{∗} = I Proof: Let Λ = diag(λ_{1}, ⋯, λ_{n}) and V = [v_{1}, ⋯, v_{n}] be the eigenvalue and

eigenvector matrices of a square matrix A:

AV = VΛ  (A.102)

and Σ = diag(σ_{1}, ⋯, σ_{n}) and U = [u_{1}, ⋯, u_{n}] be the eigenvalue and eigenvector matrices of B = R^{∗}AR, a unitary transform of A:

BU = (R^{∗}AR) U = UΣ  (A.103)

Left-multiplying R on both sides we get the eigenequation of A

RR^{∗}ARU = A(RU) = (RU)Σ  (A.104)

We see that A and B = R^{∗}AR have the same eigenvalues Σ = Λ and their eigenvector matrices are related by V = RU or U = R^{∗}V.

• Given all eigenvalues λ_{1}, ⋯, λ_{n} of a matrix A, its trace and determinant can be obtained as

tr(A) = ∑_{k=1}^{n} λ_{k},  det(A) = ∏_{k=1}^{n} λ_{k}  (A.105)

• The spectrum of an n × n square matrix A is the set of its eigenvalues {λ_{1}, ⋯, λ_{n}}. The spectral radius of A, denoted by ρ(A), is the maximum of the absolute values of the elements in the spectrum:

ρ(A) = max(|λ_{1}|, ⋯, |λ_{n}|)  (A.106)

where |z| = \sqrt{x^{2} + y^{2}} is the modulus of a complex number z = x + jy. If all eigenvalues are sorted such that |λ_{1}| ≥ ⋯ ≥ |λ_{n}| then ρ(A) = |λ_{1}| = |λ_{max}|. As the eigenvalues of A^{−1} are {1/λ_{max}, ⋯, 1/λ_{min}}, ρ(A^{−1}) = 1/|λ_{min}.

• If A and B are similar, i.e.,

B = P^{−1}AP  (A.107)

then they have the same eigenvalues.

Proof: Let Λ and V be the eigenvalue and eigenvector matrices of A:

AV = VΛ,  A = VΛV^{−1}  (A.108)

then we have

B = P^{−1}AP = P^{−1}VΛV^{−1}P = UΛU^{−1}  (A.109)

i.e., Λ and U = P^{−1}V are the eigenvalue and eigenvector matrices of B = P^{−1}AP.

• If A is Hermitian A^{∗} = Ā^{T} = A (symmetric A^{T} = A if real) (e.g., the covariance matrix of a random vector), then all of its eigenvalues λ̄_{i} = λ_{i} are real, and all of its eigenvectors are orthogonal, v_{i}^{∗}v_{j} = 0 (i ≠ j) i.e., V^{−1} = V^{∗}. And we further have

A = VΛV^{−1} = VΛV^{T} = VΛ^{1/2}Λ^{1/2}V^{T} = UU^{T}  (A.110)

where U = VΛ^{1/2}.

Proof:

Let λ and v be an eigenvalue and the corresponding eigenvector of a Hermitian matrix A = A^{∗}, i.e., Av = λv, then we have

(Av)^{∗}v = (λv)^{∗}v = λ̄v^{∗}v = λ̄ ||v||^{2}
= v^{∗}A^{∗}v = v^{∗}Av = v^{∗}λv = λ ||v||^{2}  (A.111)

i.e., λ̄ = λ is real. We also have

v_{i}^{∗}Av_{j} = v_{i}^{∗}λ_{j}v_{j} = λ_{j}v_{i}^{∗}v_{j}
= (v_{j}^{∗}Av_{i})^{∗} = (v_{j}^{∗}λ_{i}v_{i})^{∗} = λ̄_{i}v_{i}^{∗}v_{j} = λ_{i}v_{i}^{∗}v_{j}  (A.112)

i.e.,

λ_{j}v_{i}^{∗}v_{j} = λ_{i}v_{i}^{∗}v_{j}, or (λ_{i} − λ_{j}) v_{i}^{∗}v_{j} = 0  (A.113)

As λ_{i} ≠ λ_{j}, we get v_{i}^{∗}v_{j} = 0, i.e., the eigenvectors corresponding to different eigenvalues are orthogonal.

QED

When all eigenvectors are normalized v_{i}^{∗}v_{i} = ||v_{i}||^{2} = 1, they become orthonormal

⟨v_{i}, v_{j}⟩ = v_{i}^{∗}v_{j} = δ_{ij} = \{ \begin{matrix}1 & i = j \\ 0 & i ≠ j \end{matrix}  (A.114)

i.e., the eigenvector matrix V = [v_{1}, ⋯, v_{n}]^{T} is unitary (orthogonal if A is real):

V^{−1} = V^{∗} i.e. V^{∗}V = I  (A.115)

and we have

V^{−1}AV = V^{∗}AV = Λ,  (A.116)

Left and right multiplying by V and V^{∗} = V^{−1} respectively on the two sides, we get

A = VΛV^{∗} = [v_{1}, ⋯, v_{n}] [ \begin{matrix}λ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & λ_{n} \end{matrix} ] [ v_{1}^{∗} ; ⋮ ; v_{n}^{∗} ] = ∑_{i=1}^{n} λ_{i}v_{i}v_{i}^{∗}

(A.117)

This is the spectral theorem indicating that A can be written as a linear combination of n matrices v_{i}v_{i}^{∗} weighted by λ_{i} (n = 1, ⋯, n).

The significance of this property is that a linear operation y = Ax applied to vector x can be mapped to a new vector space in which the operations of the components are independent of each other. Consider

y = Ax = VΛV^{∗}x  (A.118)

Pre-multiplying V^{∗} on both sides we get

y′ = V ∗ y = ΛV^{∗}x = Λx′  (A.119)

where we have defined x′ = V^{∗}x and y′ = V^{∗}y, the unitary transform of x and y, respectively. We see that for the ith component we have the following, independent of all other components:

y′_{i} = λ_{i}x′_{i}  (A.120)

• The entries on the diagonal of an upper (or lower) triangular matrix are its

eigenvalues.

Proof: Let A be an upper triangular matrix with a_{ij} = 0 for all i > j.

The eigenvalues of A are the roots of the following homogeneous characteristic equations:

det(A − λI) = ∏_{i=1}^{n}(a_{ii} − λ) = 0  (A.121)

The first equal sign is due to the fact that A−λI is also an upper-triangular matrix, and the determinant of an upper-triangular matrix is the product of all its diagonal entries. We therefore see that each diagonal entry a_{ii}, as a root of the characteristic equation, is also an eigenvalue of A.

QED

• Similar matrices have the same eigenvalues.

Proof: Let λ and v be an eigenvalue and the corresponding eigenvector of A satisfying Av = λv, and B = P^{−1}AP be a similar matrix of A. We have A = PBP^{−1} and

Av = PBP^{−1}v = λv, i.e., BP^{−1}v = λP^{−1}v  (A.122)

In other words, λ is also an eigenvalue of B with the corresponding eigenvector P^{−1}v.

QED

• All eigenvalues of a stochastic matrix P are no greater than 1. A stochastic matrix is a matrix of which component P_{ij} is the probability for a state transition of a Markov process (chain) from s_{i} to s_{j}, i.e., 0 ≤ P_{ij} ≤ 1 and ∑_{j} P_{ij} = 1.

Proof: First, as P1 = 1, λ = 1 is one of the eigenvalues of P. Next let λ and v be an eigenvalue and the corresponding eigenvector of P, i.e., Pv = λv, and let |v_{k}| ≥ max{|v_{1}|, ⋯, |v_{n}|}. The kth row of the eigenequation is

|λv_{k}| ≤ |λ| |v_{k}| = | ∑_{j=1}^{n} P_{kj}v_{j}| ≤ ∑_{j=1}^{n} P_{kj} |v_{j}| ≤ |v_{k}| ∑_{j=1}^{n} P_{kj} = |v_{k}| (A.123)

i.e., |λ| ≤ 1.

QED

A.3.2 Generalized Eigenvalue Problem

The generalized eigenvalue problem of two symmetric matrices A = A^{T} and B = B^{T} is to find a scalar λ_{i} and the corresponding vector v_{i} for the following generalized eigen equation to hold:

Av_{i} = λ_{i}Bv_{i},  (i = 1, ⋯, n)  (A.124)

or in matrix form

AV = BVΛ  (A.125)

The eigenvalue and eigenvector matrices Λ and V can be found in the following steps.

• Solve the eigenvalue problem of B to find its diagonal eigenvalue matrix Λ_{B} and orthogonal eigenvector matrix V_{B} = (V_{B}^{T})^{−1} so that

BV_{B} = V_{B}Λ_{B},  or V_{B}^{−1}BV_{B} = V_{B}^{T}BV_{B} = Λ_{B}  (A.126)

• Left and right multiplying both sides of the second equation above by Λ^{−1/2} (whitening) we get

Λ_{B}^{−1/2}(V_{B}^{T}BV_{B})Λ_{B}^{−1/2} = Λ_{B}^{−1/2}Λ_{B}Λ_{B}^{−1/2} = I  (A.127)

We define

V′_{B} = V_{B}Λ_{B}^{−1/2}  (A.128)

and the above becomes

(V′_{B})^{T} BV′_{B} = I  (A.129)

Note that V′_{B} is not orthogonal

(V′)_{B}^{−1} = (V_{B}Λ_{B}^{−1/2})^{−1} = Λ_{B}^{1/2}V_{B}^{−1} = Λ_{B}^{1/2}V_{B}^{T} ≠ Λ_{B}^{−1/2}V_{B}^{T} = V_{B}^{T}

(A.130)

• Apply the same transform to A:

(V′_{B})^{T} AV′_{B} = (Λ_{B}^{−1/2}V_{B}^{T})A(V_{B}Λ_{B}^{−1/2}) = A′  (A.131)

Note that A′ is symmetric as well as A:

A^{′T} = (V_{B}^{′T} AV′_{B})^{T} = V_{B}^{′T} AV′_{B} = A′  (A.132)

• Diagonalize A′

As A′ is symmetric,, it can be diagonalized by its orthogonal eigenvector matrix V_{A}:

V_{A}^{T}A′V_{A} = Λ  (A.133)

i.e.,

V_{A}^{T}(Λ_{B}^{−1/2}V_{B}^{T}AV_{B}Λ_{B}^{−1/2})V_{A} = (V_{A}^{T}Λ_{B}^{−1/2}V_{B}^{T})A(V_{B}Λ_{B}^{−1/2}V_{A})
= V^{T} AV = Λ

where we have defined

V = V_{B}Λ_{B}^{−1/2}V_{A}  (A.134)

which is not orthogonal:

V^{−1} = (V_{B}Λ_{B}^{−1/2}V_{A})^{−1} = V_{A}^{−1}Λ_{B}^{1/2}V_{B}^{−1} = V_{A}^{T}Λ_{B}^{1/2}V_{B}^{T} ≠ V_{A}^{T}Λ_{B}^{−1/2}V_{B}^{T} = V^{T}

(A.135)

• This V also diagonalizes B:

V^{T} BV = (V_{B}Λ_{B}^{−1/2}V_{A})^{T} B(V_{B}Λ_{B}^{−1/2}V_{A}) = V_{A}^{T}Λ_{B}^{−1/2}(V_{B}^{T}BV_{B})Λ_{B}^{−1/2}V_{A}
= V_{A}^{T}Λ_{B}^{−1/2}Λ_{B}Λ_{B}^{−1/2}V_{A} = V_{A}^{T}V_{A} = I  (A.136)

• Now we have

\{ \begin{matrix}V^{T} AV = Λ \\ V^{T} BV = I \end{matrix}  (A.137)

Right multiplying both sides of the second equation by Λ we get V^{T} BVΛ = Λ, then equating the left-hand side to that of the first equation, we get

V^{T} AV = V^{T} BVΛ, i.e. AV = BVΛ  (A.138)

i.e., Λ and V are the eigenvalue and eigenvector matrices of the generalized eigenvalue problem. Note, however, as shown above, V is not orthogonal.

A.3.3 Rayleigh Quotient

The Rayleigh quotient R(A) of a hermitian matrix A = A^{∗} (symmetric if A is real) is defined as

R(A) = \frac{w^{∗}Aw}{w^{∗}w}  (A.139)

where w is any non-zero vector. In particular, if w = v_{i} is the ith eigenvector of A, then the Rayleigh quotient is the corresponding to eigenvalue λ_{i}:

R(A) = \frac{v_{i}^{∗}Av_{i}}{v_{i}^{∗}v_{i}} = \frac{λ_{i}v_{i}^{∗}v_{i}}{v_{i}^{∗}v_{i}} = λ_{i}  (A.140)

The Rayleigh quotient of two symmetric matrices A and B is a function of a vector w defined as:

R(w) = \frac{w^{T} Aw}{w^{T} Bw}  (A.141)

To find the optimal w corresponding to the extremum (maximum or minimum) of R(w), we find its derivative with respect to w:

\frac{d}{dw} R(w) = \frac{2Aw(w^{T} Bw) − 2Bw(w^{T} Aw)}{(w^{T} Bw)^{2}}  (A.142)

Setting it to zero we get

Aw(w^{T} Bw) = Bw(w^{T} Aw), i.e. Aw = \frac{w^{T} Aw}{w^{T} Bw} Bw = R(w)Bw = λBw

(A.143)

The second equation is the generalized eigenequation of which λ = R(w) is the eigenvalue and w the corresponding eigenvector. Solving this generalized eigenvalue problem we get the eigenvector w corresponding to the maximum or minimum eigenvalue, the Rayleigh quotient λ = R(w).

A.3.4 Normal Matrices and Diagonalizability

Theorem: The product of two unitary matrices is unitary.

Proof: Let U and V be unitary, i.e., U^{∗} = U^{−1} and V^{∗} = V^{−1}, then UV is unitary:

(UV)^{∗} = V^{∗}U^{∗} = V^{−1}U^{−1} = (UV)^{−1}  (A.144)

Theorem: Two square matrices A and B are simultaneously diagonalizable if and only if they commute.

Proof

• Let A and B be simultaneously diagonalizable by R

R^{−1}AR = Λ_{A},  R^{−1}BR = Λ_{B}  (A.145)

then

AB = (RΛ_{A}R^{−1})(RΛ_{B}R^{−1}) = (RΛ_{A}Λ_{B}R^{−1})
= (RΛ_{B}Λ_{A}R^{−1}) = (RΛ_{B}R^{−1})(RΛ_{A}R^{−1})
= BA  (A.146)

• Let A and B commute, i.e., AB = BA. Assuming u is an eigenvector of A corresponding to eigenvalue λ, i.e., Au = λu, then

ABu = B(Au) = λBu  (A.147)

we see that Bu is also an eigenvector of A corresponding to the same eigenvalue λ, i.e., Bu must be a scaled version of u (in the same 1-D space): Bu = γu, i.e., u is also an eigenvector of B.

Theorem: A matrix is normal if and only if it is unitarily diagonalizable. Proof

• If A is unitarily diagonalizable:

AU = UΛ,  U^{−1}AU = U^{∗}AU = Λ,  A = UΛU^{∗}  (A.148)

where U^{∗} = U^{−1} is unitary and Λ is a diagonal matrix satisfying Λ^{∗}Λ = ΛΛ^{∗}, then A is normal:

AA^{∗} = (UΛU^{∗})(UΛU^{∗})^{∗} = (UΛU^{∗})(UΛ^{∗}U^{∗}) = UΛΛ^{∗}U^{∗}
= UΛ^{∗}ΛU^{∗} = (UΛ^{∗}U^{∗})(UΛU^{∗}) = A^{∗}A

• If A is normal, then it is diagonalizable by a unitary matrix. First we show any matrix A can be written as

A = B + iC  (A.149)

where

B = \frac{1}{2}(A + A^{∗}) = B^{∗}, C = −\frac{1}{2}(A − A^{∗}) = C^{∗},  (A.150)

are both Hermitian, and diagonalizable by a unitary matrix. As A is normal, we have

0 = AA^{∗} − A^{∗}A = (B + iC)(B − iC) − (B − iC)(B + iC) = 2i(CB − BC)

(A.151)

We see that CB = BC, i.e., B and C commute, and they can be simultaneously diagonalized by some unitary matrix U:

U^{∗}BU = Λ_{B},  U^{∗}CU = Λ_{C},  (A.152)

and so can A = B + iC:

U^{∗}AU = U^{∗}(B + iC)U = Λ_{B} + iΛ_{C} = Λ_{A}  (A.153)

Q.E.D.

If A is normal, i.e., it commutes with its conjugate transpose A^{∗}A = AA^{∗}, then A and A^{∗} can simultaneously diagonalized by their unitary eigenvector matrix U:

A = UΛU^{∗}, A^{∗} = U\overline{Λ}U^{∗}  (A.154)

A.3.5 Singular Value Decomposition

The eigenvalue decomposition of a square matrix A = VΛV^{−1} is of great importance and widely used. However, for an m × n non-square matrix A, no eigenvalues and eigenvector exist. In this case, we can still find its singular values and the corresponding left and right singular vectors, and then carry out singular value decomposition (SVD).

Theorem: An m × n matrix A of rank r ≤ min(m, n) can be diagonalized by two orthogonal matrices U_{m×m} and V_{n×n}:

U^{T} AV = Σ  or A = UΣV^{T}  (A.155)

where

• U = [u_{1}, …, u_{m}] is an m×m matrix composed of the orthogonal eigenvectors of the symmetric matrix AA^{T}, also called the left singular vectors of A:

U^{T} (AA^{T})U = Λ  or AA^{T} = UΛU^{T}  (A.156)

where Λ = diag(λ_{1}, ⋯, λ_{n}) is the e igenvalue matrix of AA^{T}.

• V = [v_{1}, …, v_{n}] is an n × n matrix composed of the orthogonal eigenvectors of the symmetric matrix A^{T} A, also called the right singular vectors of A:

V^{T} (A^{T} A)V = Λ  or A^{T} A = VΛV^{T}  (A.157)

Note that A^{T} A and AA^{T} have the same eigenvalue matrix Λ (see proof below).

• Σ = Λ^{1/2} is an m × n diagonal matrix of r non-zero values σ_{i} = \sqrt{λ_{i}} (i = 1, …, r), the square roots of the eigenvalues of AA^{T} or A^{T} A, defined as the singular values of A:

Λ^{1/2} = [ \begin{matrix}√ & \overline{λ_{1}} \\ & . & . & . \\ √ & \overline{λ_{r}} \end{matrix}  ] = [ \begin{matrix}σ_{1} \\ & . & . & . \\ & & & & σ_{r} \end{matrix}  ] = Σ (A.158)

Proof:

Pre-multiplying A on both sides of the eigenequation of A^{T} A:

(A^{T} A)V = VΛ  (A.159)

we get

(AA^{T})AV = AVΛ,  (A.160)

Note that A^{T} A and AA^{T} have the same eigenvalue matrix Λ, and AV is the eigenvector matrix of AA^{T}. As the columns (or rows) of AV are not normalized, AV is not orthogonal:

(AV)^{T} (AV) = V^{T} A^{T} AV = V^{T} VΛ = Λ ≠ I  (A.161)

If we post-multiply AV by Λ^{−1/2}, its ith column is scaled by 1/\sqrt{λ_{i}} (i = 1, ⋯, n) and thereby normalized, the resulting matrix becomes orthogonal:

(AVΛ^{−1/2})^{T} (AVΛ^{−1/2}) = Λ^{−1/2}V^{T} A^{T} AVΛ^{−1/2}
= Λ^{−1/2}V^{T} VΛΛ^{−1/2} = Λ^{−1/2}ΛΛ^{−1/2} = I

We denote this matrix composed of normalized orthogonal eigenvectors of AA^{T} by

AVΛ^{−1/2} = U  (A.162)

Post-multiplying (VΛ^{−1/2})^{−1} = Λ^{1/2}V^{−1} on both sides, we get the SVD of A:

A = UΛ^{1/2}V^{−1} = UΛ^{1/2}V^{T} = UΣV^{T}  (A.163)

Q.E.D.

The SVD is illustrated graphically below for the two cases of m < n and m > n:

The component form of AVΛ^{−1/2} = U above is

Av_{i} \frac{1}{\sqrt{λ_{i}}} = u_{i} (i = 1, ⋯, n)  (A.164)

relating the eigenvectors v_{i} of A^{T} A to the eigenvectors u_{i} of AA^{T}. We further consider some special cases:

• If A^{T} = A^{−1} is unitary, i.e., A^{T} A = AA^{T} = I, its eigenvalue matrix is Λ = I, and all singular values are unity σ_{i} = 1 for all i = 1, ⋯, n.

• If A^{T} = A is symmetric, i.e., AA^{T} = A^{T} A = A^{2}, and its singular value decomposition is A = UΣV^{T}, i.e.,

AA^{T} = UΣ^{2}U^{T} , A^{T} A = VΣ^{2}V  (A.165)

On the other hand, the eigenvalue decomposition of A^{2} = A^{T} A = AA^{T} can be written as:

A^{2} = WΛ^{2}W  (A.166)

where Λ is the eigenvalue matrix of A. We see that Λ^{2} = Σ^{2}, i.e., the singular values are the absolute values of its eigenvalues σ_{i} = \sqrt{λ_{i}^{2}} = |λ_{i}| (i = 1, ⋯, n), and the left and right eigenvectors are the same as the eigenvectors U = V = W.

• If A is normal satisfying A^{T} A = AA^{T} (e.g., symmetric, Hermitian, unitary, etc.), both A and A^{T} can be simultaneously diagonalized by their unitary eigenvector matrix U = V, i.e.,

Au_{i} = λ_{i}u_{i}, Au_{i} = \overline{λ_{i}}u_{i}, (i = 1, ⋯, n)  (A.167)

or in matrix form

A = UΛU^{T} , A^{T} = U\overline{Λ}U^{T}  (A.168)

we have

AA^{T} = (UΛU^{T})(U\overline{Λ}U^{T}) = UΛ\overline{Λ}U^{T} = U|Λ|^{2}U^{T}  (A.169)

i.e., the singular values of any norm matrix A are simply the modulus of its eigenvalues σ_{i}(A) = |λ_{i}(A)| (i = 1, ⋯, n).

The m × n matrix A = UΣV^{T} can be considered as a linear transformation that converts a vector x ∈ ℂ^{n} to another vector y ∈ ℂ^{m} in three steps:

y = Ax = UΣV^{T} x  (A.170)

1. Rotate vector x by the orthogonal matrix V^{T}:

y_{1} = V^{T} x.  (A.171)

2. Scale each dimension Y_{k} of y_{1} by a factor of σ_{k} (k = 1, …, r):

y_{2} = Σ y_{1} = ΣV^{T} x.  (A.172)

3. Rotate vector y_{2} by the orthogonal matrix U:

y = Uy_{2} = UΣV^{T} x = Ax.  (A.173)

The figure below illustrates the transformation of the three vertices of a triangle in 2-D space by a matrix A = UΣV^{∗}, which first rotates the vertices by 45 degrees CCW, scale horizontally and vertically by a factor of 3 and 2, respectively, and then rotate CW by 30 degrees.

The SVD has many applications, two of which are considered below.

The SVD equation

U^{T} AV = Λ^{1/2} = Σ  (A.174)

can be considered as the forward SVD transform. Pre-multiplying U and postmultiplying V^{T} on both sides, we get the inverse transform:

A = UΣV^{T} = [u_{1}, · · · · · · · · · , u_{m}] [ \begin{matrix}σ_{1} \\ & . & . & . \\ & & & & σ_{r} \\ & & & & & 0 \\ & & & & & & . & . & . \\ & & & & & & & & & 0 \end{matrix} ] [ v_{1}^{T} ; ⋮ ; ⋮ ; ⋮ ; v_{n}^{T} ] = ∑_{i=1}^{r} σ_{i}[u_{i}v_{i}^{T}]

(A.175)

by which the original matrix A is represented as a linear combination of r matrices [u_{i}v_{i}^{T}] weighted by the singular values \sqrt{λ_{i}} (i = 1, …, r). We can rewrite both the forward and inverse SVD transform as a pair of forward and inverse transforms:

\{ \begin{matrix}Σ = Λ^{1/2} = U^{T} AV \\ A = UΛ^{1/2}V^{T} = UΣV^{T} . \end{matrix}  (A.176)

This result indicate that the SVD can be used for data compression, if the 2-D data stored in an m × n matrix A can be approximated by keeping in the summation above only k ≪ r ≤ min(m, n) terms corresponding to the k greatest singular values. The amount of data can be significantly reduced from m × n to no more than k(m + n + 1).

Also, the pseudo-inverse of any matrix (non-square, not full-rank) can be found given its SVD

A = UΛ^{1/2}V^{T} = UΣV^{T}  (A.177)

as

A^{−} = (V^{T})^{−1}Σ^{−}U^{−1} = VΣ^{−}U^{T}  (A.178)

where Σ^{−} is pseudo-inverse of Σ, composed of the reciprocals 1/σ_{k} of the R singular values along the diagonal. The pseudo-inverse matrix is needed to find optimal solution of an inconsistent linear equation system.

A.4 Vector and Matrix Calculations

A.4.1 Vector Norms

The norm of a vector x, denoted by ||x||, can be intuitively interpreted as its “size”. For example, the norm of a real number x ∈ ℝ in the 1-D real space is its absolute value |x|, or its distance to the origin, and the norm of a complex number z = x + jy ∈ ℂ is its modulus |z| = \sqrt{x^{2} + y^{2}}, its Euclidean distance to the origin. Here is the most general definition of a vector norm:

Definition: The norm of a vector x ∈ V in vector space V is a real nonnegative value representing the length or magnitude of the vector. Specifically, the norm of x ∈ V satisfy the following three conditions:

• Positivity:

||x|| ≥ 0, ||x|| = 0 iff x = 0  (A.179)

• Homogeneity:

||ax|| = |a| ||x||, a ∈ ℝ or ℂ  (A.180)

• Triangle inequality:

||x + y|| ≤ ||x|| + ||y||  (A.181)

The triangle inequality can also be expressed in alternative forms. If we define z = −y, the triangle inequality becomes

||x − z|| ≤ ||x|| + ||z||  (A.182)

If we further define u = x − z, i.e., z = x − u, then the above becomes

||u|| ≤ ||x|| + ||x − u||, i.e. ||u|| − ||x|| ≤ ||x − u|| = ||u − x|| (A.183)

Combining the two results above, we get

||x|| − ||y|| ≤ ||x − y|| ≤ ||x|| + ||y||  (A.184)

The first equality holds if x and y are in the same direction, the second equality holds if they are in opposite direction. More specially when y = 0, both equalities hold.

The p-norms of an n-D vector x = [x_{1}, ⋯, x_{n}]^{T} and a function f(x) are defined as:

||x||_{p} = (∑_{i=1}^{n} |x_{i}|^{p})^{1/p} ,  ||f(x)||_{p} = (∫ |f(x)|^{p} dx)^{1/p}  (A.185)

The p-norm satisfies the three requirements in the definition of vector norm. The first two are trivially obvious, while the third one happens to be Minkowski’s inequality (see appendix):

||x + y||_{p} ≤ ||x||_{p} + ||y||_{p}  (A.186)

The p-norms corresponding to p = 1, p = 2, and p = ∞ are most commonly used:

• p = 1, the absolute sum of all n elements:

||x||_{1} = ∑_{i=1}^{n} |x_{i}|  (A.187)

• p = 2, the Euclidean norm:

||x||_{2} = (x^{T} x)^{1/2} = \sqrt{∑_{i=1}^{n} |x_{i}|^{2}}  (A.188)

• p = ∞, the maximum absolute value of all n elements:

||x||_{∞} = (∑_{i=1}^{n} |x_{i}|^{∞})^{1/∞} = max{|x_{1}|, ⋯, |x_{n}|}  (A.189)

Out of the three p-norms, the Euclidean 2-norm is the only one that is unitary invariant, i.e., it is conserved or invariant under any unitary transform y = Rx, where R is a unitary matrix satisfying R^{−1} = R^{∗}:

||y||_{2}^{2} = y^{∗}y = (Rx)^{∗}(Rx) = x^{∗} (R^{∗}R) x = x^{∗}x = ||x||_{2}^{2}  (A.190)

i.e., the length of the vector is not changed by any unitary transform (such as rotation, when R is a rotation matrix).

Definition: Two norms ||x||_{a} and ||x||_{b} are equivalent if there exist two positive real constants c and C so that

c ||x||_{b} ≤ ||x||_{a} ≤ C ||x||_{b}  (A.191)

Note that this relationship can also be written as

\frac{1}{C} ||x||_{a} ≤ ||x||_{b} ≤ \frac{1}{c} ||x||_{a}  (A.192)

Theorem: All different norms x are equivalent.

Proof

• We first show that equivalence is transitive, i.e., if both ||x||_{a} and ||x||_{b} are equivalent to ||x||_{1}:

c ||x||_{1} ≤ ||x||_{a} ≤ C ||x||_{1}, and d ||x||_{1} ≤ ||x||_{b} ≤ D ||x||_{1}, (A.193)

then they are equivalent to each other.

From the first equation we get ||x||_{1} ≤ ||x||_{a}/c and ||x||_{1} ≥ ||x||_{a}/C. Substituting these into the right and left hand sides of the second equation respectively yields:

\frac{d}{C} ||x||_{a} ≤ ||x||_{b} ≤ \frac{D}{c} ||x||_{a},  (A.194)

i.e., ||x||_{a} and ||x||_{b} are equivalent.

• If we can further show that an arbitrary norm ||x|| is equivalent to ||x||_{1}, i.e.,

c ||x||_{1} ≤ ||x|| ≤ C ||x||_{1}  (A.195)

then according to the transitivity property shown above, all norms are equivalent.

If x = 0, it is obvious that the equalities above hold ||x|| = ||x||_{1} = 0. If x ≠ 0, ||x||_{1} > 0, we can define a normalized vector u = x/||x||_{1} with ||u||_{1} = 1, so that the relationship for the equivalence above can be written in terms of u as

c = c||u||_{1} ≤ ||u|| ≤ C||u||_{1} = C  (A.196)

i.e., to prove that all norms are equivalent all we need to show is that ||u|| is bounded both from above and below, as we will do below.

• We now prove that an arbitrary norm f(x) = ||x|| is a continuous function over ||x||_{1}, i.e., for any ϵ > 0, there exists a δ > 0 so that

if ||x − x′||_{1} < δ, then ||x|| − ||x′|| ≤ ϵ  (A.197)

Both x and x′ can be expressed in terms of a basis {b_{1}, ⋯, b_{n}} that spans the space:

x = ∑_{i=1}^{n} a_{i}b_{i}, x′ = ∑_{i=1}^{n} a′_{i}b_{i}  (A.198)

then we have

||x−x′|| = || ∑_{i=1}^{n}(a_{i}−a′_{i})b_{i}|| ≤ ∑_{i=1}^{n} |a_{i}−a′_{i}| ||b_{i}|| ≤ ∑_{i=1}^{n} |a_{i}−a′_{i}| max_{i} ||b_{i}|| = ||x−x′||_{1} max_{i} ||b_{i}||

(A.199)

Now consider the alternative form of the triangle inequality of ||x||:

||x|| − ||x′|| ≤ ||x − x′|| ≤ ||x − x′||_{1} max_{i} ||b_{i}||  (A.200)

If we let max_{i} ||b_{i}|| = ϵ/δ, the above becomes

||x|| − ||x′|| ≤ ||x − x′||_{1} \frac{ϵ}{δ}  (A.201)

indicating that for any given ϵ, we can choose δ = ϵ/ max_{i} ||b_{i}|| so that if ||x−x′||_{1} ≤ δ, then ||x||−||x′|| ≤ ϵ, i.e., the norm ||x|| is indeed continuous over ||x||_{1}.

• Finally, according to the extreme value theorem, a continuous function, such as f(x) = ||x||, defined over a compact (closed and bounded) set, such as the unit sphere ||u||_{1} = 1 in the n-D space, must have its maximum and minimum values:

c = min_{||u||_{1}=1} ||u||, C = max_{||u||_{1}=1} ||u||  (A.202)

i.e., c ≤ ||u|| ≤ C, which is what we need to prove.

Q.E.D.

The three p-norms are equivalent:

||x||_{∞} ≤ ||x||_{1} ≤ n||x||_{∞}
||x||_{∞} ≤ ||x||_{2} ≤ \sqrt{n}||x||_{∞}
||x||_{2} ≤ ||x||_{1} ≤ \sqrt{n}||x||_{2}

The distance between two vectors x, y ∈ V is fined as the norm of their difference d(x, y) = ||x − y||. In particular, when y = 0, the distance ||x − 0|| = ||x|| between x and the origin 0 of the space is the norm of x. Specifically, d(x, y) can be defined based on the p-norm:

d_{p}(x, y) = ||x − y||_{p} = (∑_{i=1}^{n} |x_{i} − y_{i}|^{p})^{1/p} , (1 ≤ p ≤ ∞)  (A.203)

Specially, when

• p = 1, d_{1}(x, y) is the city block (Manhattan) distance:

d_{1}(x, y) = ||x − y||_{1} = ∑_{i=1}^{n} |x_{i} − y_{i}|  (A.204)

• p = 2, d_{2}(x, y) is the Euclidean distance:

d_{2}(x, y) = ||x − y||_{2} = \sqrt{∑_{i=1}^{n} |x_{i} − y_{i}|^{2}}  (A.205)

• p = ∞, d_{∞}(x, y) is the Chebyshev distance:

d_{∞}(x, y) = ||x − y||_{∞} = max_{i} |x_{i} − y_{i}|  (A.206)

The three unit “circles” or “spheres”, are formed by all points x of unity norm ||x||_{p} = 1 with unity distance to the origin (blue, black, and red for d_{∞}, d_{2}, and d_{1}, respectively).

A.4.2 Matrix Norms

The norm of an m × n matrix A is any function that maps A to a real number ||A|| that satisfies the following required properties:

• Positivity:

||A|| ≥ 0, ||A|| = 0 iff A = 0  (A.207)

• Homogeneity:

||aA|| = |a| ||A||  (A.208)

• Triangle inequality:

||A + B|| ≤ ||A|| + ||B||, or ||A|| − ||B|| ≤ ||A − B|| (A.209)

The second expression can be obtained by replacing A by A − B in the first expression.

In addition to the three required properties for matrix norms, some of them also satisfy these additional properties (not necessarily satisfied by other matrix norms):

• Subordinance:

||Ax|| ≤ ||A|| · ||x||  (A.210)

• submultiplicativity:

||AB|| ≤ ||A|| · ||B||  (A.211)

We now consider some commonly used matrix norms:

• The element-wise norms

If we treat the m×n elements of A are the elements of an mn-dimensional vector, then the p-norm of this vector can be used as the p-norm of A:

||A||_{p} = {∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{p}}^{1/p}  (A.212)

Specially, we consider the following three cases for p = 1, 2, ∞.

– ||A||_{1} is the absolute sum of all elements of A:

||A||_{1} = ∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|  (A.213)

– ||A||_{∞} is the maximum norm, the maximum absolute value among all mn elements of A:

||A||_{∞} = max{|a_{ij}|, 1 ≤ i ≤ m, 1 ≤ j ≤ n}  (A.214)

– ||A||_{2} is the Frobenius norm

||A||_{2} = ||A||_{F} = \sqrt{∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{2}} = \sqrt{tr(A^{∗}A)} = \sqrt{∑_{i=1}^{R} λ_{i}} = \sqrt{∑_{i=1}^{R} σ_{i}^{2}}

(A.215)

where r ≤ min(m, n) is the rank of A, λ_{i} = σ_{2}^{2} is the ith non-zero eigenvalues of the positive, semi-definite matrix A^{∗}A, and σ_{i} = \sqrt{λ_{i}} the ith singular value of A (i = 1, ⋯, r). (This Frobenius norm ||A||_{2} is implemented in Matlab by the function norm(A,'fro').)

The Frobenius norm of a unitary (orthogonal if real) matrix R satisfying R^{∗} = R^{−1} or R^{∗}R = R^{−1}R = I is:

||R||_{F}^{2} = tr(R^{∗}R) = trI = n  (A.216)

The Frobenius norm is the only one out of the above three matrix norms that is unitary invariant, i.e., it is conserved or invariant under a unitary transformation (such as a rotation) B = R^{∗}AR:

||B||_{F}^{2} = tr (B^{∗}B) = tr [(R^{∗}AR)^{∗} (R^{∗}AR)] = tr (R^{∗}A^{∗}RR^{∗}AR)
= tr (R^{∗}A^{∗}AR) = tr (A^{∗}A) = ||A||_{F}^{2} = ||A||_{F}^{2}

where we have used the property of the trace tr (AB) = tr (BA).

• The natural matrix norms

The natural norm ||A|| of matrix A induced by (subordinate to) the vector norm ||x|| is defined as

||A|| = sup_{||x||=1}{||Ax||} = sup_{x≠0}{||Ax||/||x||}  (A.217)

Specially the norm of the identity matrix A = I is

||I|| = sup_{||x||=1}{||Ix||} = sup_{||x||=1}{||x||} = 1  (A.218)

We now prove that the induced matrix norm satisfies all five properties given above. (Recall |a + b| ≤ |a| + |b|, |ab| ≤ |a| |b|.) 1.Positivity: ||A|| > 0 if A ≠ 0, this is trivially obvious. 2.Homogeneity: ||aA|| = |a| ||A||

||aA|| = sup_{||x||=1}{||aAx||} = sup_{||x||=1}{|a| ||Ax||} = |a| sup_{||x||=1}{||Ax||} = |a| ||A||

(A.219)

3.Triangle inequality: ||A + B|| ≤ ||A|| + ||B||

||A + B|| = sup_{||x||=1}{||(A + B)x||} = sup_{||x||=1}{||Ax + Bx||} ≤ sup_{||x||=1}{||Ax|| + ||Bx||}
≤ sup_{||x||=1}{||Ax||} + sup_{||x||=1}{||Bx||} = ||A|| + ||B||

4.Subordinance: ||Ax|| ≤ ||A|| ||x||

||A|| ||x|| = ||x|| sup_{y}{||Ay||/||y||} ≥ ||x|| ||Ay||/||y||  (A.220)

As y is arbitrary, we let y = x and get

||A|| ||x|| ≥ ||Ax||  (A.221)

In particular, if vector x is the eigenvector corresponding to the greatest eigenvalue λ_{max} of A^{∗}A:

(A^{∗}A) x = λ_{max} x = ||A||^{2} x  (A.222)

then the equality of the subordinance property holds. To see this, consider

||Ax||^{2} = ⟨(Ax)^{∗}, (Ax)⟩ = x^{∗}(A^{∗}A)x = x^{∗}||A||^{2}x = ||A||^{2}||x||^{2}

(A.223)

taking square root on both sides we get ||Ax|| = ||A|| ||x||. 5.Submultiplicativity: ||AB|| ≤ ||A|| ||B||

||AB|| = sup_{||x||=1}{||ABx||} ≤ sup_{||x||=1}{||A|| ||Bx||} = ||A|| sup_{||x||=1}{||Bx||} = ||A|| ||B||

(A.224)

In particular, if A = cB^{∗} (i.e., A and B are linearly dependent), then the equality of the submultiplicativity property holds. To see this, consider

||B|| = \sqrt{λ_{max}(B^{∗}B)}, ||A|| = |c|\sqrt{λ_{max}(BB^{∗})} = |c|\sqrt{λ_{max}(B^{∗}B)} (A.225) and

||AB|| = |c| ||B^{∗}B|| = |c| \sqrt{λ_{max}((B^{∗}B)(B^{∗}B))} = |c| \sqrt{λ_{max}(B^{∗}B)^{2}} = |c| λ_{max}(B^{∗}B)

(A.226)

i.e., ||AB|| = ||A|| ||B||. Consider in particular the natural matrix norms ||A||_{p} corresponding to

the three vector p-norms ||x||_{p} with p = 1, 2, ∞:

– When p = 1, ||A||_{1} (norm(A,1) in Matlab) is the maximum absolute column sum:

||A||_{1} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|  (A.227)

Proof: The 1-norm of vector x is ||x||_{1} = ∑_{j=1}^{n} |x_{j}|, we have

||Ax||_{1} = ∑_{i=1}^{m} | ∑_{j=1}^{n} a_{ij}x_{j}| ≤ ∑_{i=1}^{m} ∑_{j=1}^{n} (|a_{ij}| |x_{j}|) = ∑_{j=1}^{n} (|x_{j}| ∑_{i=1}^{m} |a_{ij}|)

≤ ( max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|) (∑_{j=1}^{n} |x_{j}|) = ||A||_{1} ||x||_{1}

Assuming the kth column of A has the maximum absolute sum and x is normalized (as required in the definition) with ||x|| = 1, we have

||A||_{1} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}| = ∑_{i=1}^{m} |a_{ik}|  (A.228)

and

||Ax||_{1} ≤ ||A||_{1} ||x||_{1} = ∑_{i=1}^{m} |a_{ik}|  (A.229)

Now we show that the equality of the above can be achieved, i.e., ||Ax||_{1} is maximized, if we choose x = e_{k} = [0, ⋯, 1, ⋯, 0]^{T}, the kth unit vector (normalized):

||Ax||_{1} = ∑_{i=1}^{m} | ∑_{j=1}^{n} a_{ij}x_{j}| = ∑_{i=1}^{m} |a_{ik}| = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}| (A.230)

i.e., x = e_{k} is the vector among all other normalized vectors that maximizes ||Ax|| as required in the definition, and the resulting maximum ||A||_{1} is indeed max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|. We therefore have

||A||_{1} = sup_{||x||_{1}=1}{||Ax||_{1}} = max_{1≤j≤n} ∑_{i=1}^{m} |a_{ij}|  (A.231)

– When p = ∞, ||A||_{∞} (norm(A,inf) in Matlab) is the maximum absolute row sum:

||A||_{∞} = max_{1≤i≤m} ∑_{j=1}^{n} |a_{ij}|  (A.232)

Proof: When p = ∞, x is normalized if ||x||_{∞} = max{x_{1}, ⋯, x_{n}} =

1. The norm of vector Ax is:

||Ax||_{∞} = max {∑_{j=1}^{n} |a_{ij}x_{j}|, ⋯, ∑_{j=1}^{n} |a_{mj}x_{j}|}  (A.233)

which can be maximized by any normalized vector with ||x||_{∞} = max{x_{1}, ⋯, x_{n}} = 1 to become

||Ax||_{∞} = max {∑_{j=1}^{n} |a_{ij}|, ⋯, ∑_{j=1}^{n} |a_{mj}|}  (A.234)

We therefore have

||A||_{∞} = sup_{||x||_{∞}=1}{||Ax||_{∞}} = max_{i} ∑_{j=1}^{n} |a_{ij}|  (A.235)

For example, for any stochastic matrix P satisfying ∑_{j} P_{ij} = 1, ||P||_{∞} = 1.

– When p = 2, ||A||_{2} (norm(A,2) or norm(A) in Matlab), also called the spectral norm, is the greatest singular value of A, square root of the greatest eigenvalue of A^{∗}A, i.e., its spectral radius ρ(A^{∗}A) = λ_{max}(A^{∗}A):

||A||_{2} = σ_{max}(A) = \sqrt{λ_{max}(A^{∗}A)} = \sqrt{ρ(A^{∗}A)}  (A.236)

where λ_{max}(A^{∗}A) = σ_{max}^{2}(A) is the maximal eigenvalue of A^{∗}A. Proof: First we consider the eigenequation of A^{∗}A:

A^{∗}Av = λ_{i}v_{i}, (i = 1, ⋯, n) or A^{∗}AV = VΛ, or A^{∗}A = VΛV^{∗}

(A.237)

where

Λ = diag(λ_{1}, ⋯, λ_{n}), and V = [v_{1}, ⋯, v_{n}]  (A.238)

are the eigenvector and eigenvalue matrices of A^{∗}A. As A^{∗}A is symmetric and positive definite, all of its eigenvalues are real and positive, and assumed to be sorted

λ_{1} = λ_{max} ≥ λ_{2} ≥ ⋯ ≥ λ_{n} = λ_{min} ≥ 0  (A.239)

and all corresponding eigenvectors are orthogonal and assumed to be normalized, i.e., v_{i}^{∗}v_{j} = δ_{ij}, or V^{∗} = V^{−1} is a unitary (orthogonal if real) matrix.

Next, consider the p = 2 norm of Ax:

||Ax||_{2} = [(Ax)^{∗}(Ax)]^{1/2} = (x^{∗}A^{∗}Ax)^{1/2} = (x^{∗}VΛV^{∗}x)^{1/2}
= (y^{∗}Λy)^{1/2} = \sqrt{∑_{i=1}^{n} y_{i}^{2}λ_{i}}

Here y = V^{∗}x is a unitary transform of x with the 2-norm ||y||_{2} = ||x||_{2} conserved. The right-hand side of the equation above is a weighted average of the n eigenvalues λ_{1}, ⋯, λ_{n}, which is maximized if they are weighted by a normalized vector y = [1, 0, ⋯, 0]^{T} with ||y||_{2} = 1, by which the greatest eigenvalue λ_{max} is maximally weighted while all others are weighted by 0:

||Ax||_{2} ≤ \sqrt{λ_{max}}  (A.240)

As ||x||_{2} = ||y||_{2} = 1, we have

||A||_{2} = sup_{||x||_{2}=1}\{||Ax||_{2}\} = \sqrt{λ_{max}}  (A.241)

Out of the three matrix norms defined above, the spectral norm is the only one that is unitary invariant, i.e., it is conserved or invariant under any unitary transform B = R^{∗}AR:

||B||_{2}^{2} = λ_{max} (B^{∗}B) = λ_{max} [(R^{∗}AR)^{∗} (R^{∗}AR)] = λ_{max} (R^{∗}A^{∗}RR^{∗}AR)
= λ_{max} (R^{∗}A^{∗}AR) = λ_{max} (A^{∗}A) = ||A||_{2}^{2}

Here we have used the fact that the eigenvalues and eigenvectors are invariant under the unitary transform.

Example A.3

A = [ \begin{matrix}3 & −6 & 2 \\ 2 & 5 & 1 \\ −3 & 2 & 2 \end{matrix} ] ,  B = [ \begin{matrix}3 & 2 & 1 \\ 2 & −3 & 0 \\ 1 & 0 & −1 \end{matrix} ]  (A.242)

||A||_{1} = max{|3| + |2| + | − 3|, | − 6| + |5| + |2|, |2| + |1| + |2|} = max{8, 13, 5} = 13
||A||_{∞} = max{|3| + | − 6| + |2|, |2| + |5| + |1|, | − 3| + |2| + |2|} = max{11, 8, 7} = 11

The eigenvalues of A^{∗}A are

λ_{1} = 69.353, λ_{2} = 17.967, λ_{3} = 8.680  (A.243)

The singular values of A are

σ_{1} = \sqrt{λ_{1}} = 8.328, σ_{2} = \sqrt{λ_{2}} = 4.239, σ_{3} = \sqrt{λ_{3}} = 2.946 (A.244)

The spectral norm of A is

||A||_{2} = σ_{max} = \sqrt{λ_{max}} = 8.328  (A.245)

The eigenvector corresponding to greatest eigenvalue λ_{1} is x = [0.285, −0.957, 0.057]^{T}, which satisfies the equality ||Ax|| = ||A|| ||x||.

• The Schatten norms

The Shatten norm is defined based on the singular values σ_{i} of A or the eigenvalues λ_{i} = σ^{2} of A^{∗}A:

||A||_{p} = (∑_{i=1}^{R} σ_{i}^{p})^{1/p} = (∑_{i=1}^{R} (\sqrt{λ_{i}})^{p})^{1/p}  (A.246)

In particular, consider three common p values:

– p = 1 is the nuclear or trace norm:

||A||_{1} = ∑_{i=1}^{R} σ_{i} = ∑_{i=1}^{R} \sqrt{λ_{i}} = tr (\sqrt{A^{∗}A})  (A.247)

– p = 2 same as the Frobenius norm:

||A||_{2} = \sqrt{∑_{i=1}^{R} σ_{i}^{2}} = \sqrt{∑_{i=1}^{R} λ_{i}} = ||A||_{F}  (A.248)

– p = ∞ same as the spectral norm (the induced 2-norm), the spectral radius of A^{∗}A.

||A||_{∞} = max\{σ_{1}, ⋯, σ_{n}\} = σ_{max} = \sqrt{λ_{max}}  (A.249)

As the eigenvalues and eigenvectors of A^{∗}A are invariant under unitary transform, the Schatten norms are unitary invariant as well.

All matrix norms defined above are equivalent according to the theorem previously discussed.

• The Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{R} λ_{i}} and the induced 2-norm ||A||_{2} = \sqrt{λ_{max}} are equivalent:

||A||_{2} ≤ ||A||_{F} ≤ \sqrt{R} ||A||_{2}  (A.250)

The equality on the left holds when all eigenvalues λ_{i} but one are zero, and the equality on the right holds when all λ_{i} are the same.

• The Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{R} λ_{i}} and the Schatten 1-norm ||A||_{1} = ∑_{i=1}^{R} \sqrt{λ_{i}} are equivalent:

||A||_{F} ≤ ||A||_{1} ≤ \sqrt{R} ||A||_{F}  (A.251)

The equality on the left holds when all eigenvalues λ_{i} but one are zero, and the equality on the right holds when all λ_{i} are the same.

• The element-wise maximum norm ||A||_{max} = max_{i,j}{|a_{ij}|} and the Frobenius norm ||A||_{F} = \sqrt{∑_{i=1}^{m} ∑_{j=1}^{n} |a_{ij}|^{2}} are equivalent:

||A||_{max} ≤ ||A||_{F} ≤ \sqrt{mn} ||A||_{max}  (A.252)

The equality on the left holds when all elements λ_{i} but one are zero, and the equality on the right holds when all elements are the same.

•

\frac{1}{\sqrt{n}}||A||_{∞} ≤ ||A||_{2} ≤ \sqrt{m}||A||_{∞}  (A.253)

Proof: Define an n-D vector e = [1, ⋯, 1]^{T}, then the greatest absolute row sum of A is

||A||_{∞} = max_{i} ∑_{j=1}^{n} |a_{ij}| = ||Ae||_{∞} ≤ ||Ae||_{2} ≤ ||A||_{2} ||e||_{2} = \sqrt{n}||A||_{2}

(A.254)

i.e.,

\frac{1}{\sqrt{n}}||A||_{∞} ≤ ||A||_{2}  (A.255)

•

\frac{1}{\sqrt{m}}||A||_{1} ≤ ||A||_{2} ≤ \sqrt{n}||A||_{1}  (A.256)

Theorem: The spectral radius of a matrix is bounded by its matrix norm: ρ(A) ≤ ||A||

Proof: Let λ and u be an eigenvalue and the corresponding normalized eigenvector of a square matrix A, i.e., Au = λu and ||u|| = 1. As the induced norm, ||A|| satisfies

||A|| = sup_{||x||=1}{||Ax|} ≥ ||Au|| = ||λu|| = |λ| ||u|| = |λ|  (A.257)

But λ is an arbitrary eigenvalue, it can be the one with the maximum absolute value |λ|, i.e., the spectral radius ρ(A). We therefore have ||A|| ≥ ρ(A).

Theorem:

ρ(A) ≤ ||A^{k}||^{1/k}  (A.258)

Proof: Let λ and v by the eigenvalue and the corresponding eigenvector of A respectively, i.e.,

Av = λv, and A^{k}v = λ^{k}v  (A.259)

Taking norm on both sides we get

||λ^{k}v|| = |λ|^{k}||v|| = ||A^{k}v|| ≤ ||A^{k}|| ||v||  (A.260)

Dividing both sides by ||v|| ≠ 0 we get

λ^{k} ≤ ||A^{k}||, i.e. λ ≤ ||A^{k}||^{1/k}  (A.261)

Theorem: A square matrix A is convergent, i.e., lim_{n→∞} A^{n} = 0, if and only if ρ(A) < 1.

A.4.3 Vector and Matrix Differentiation

A multivariate function f(x) = f(x_{1}, ⋯, x_{n}) can be considered as a hypersurface in an n + 1 dimensional space defined over an n-D space, i.e., corresponding to every point x = [x_{1}, ⋯, x_{n}]^{T} in the n-D space, there exists a unique value f(x) (real or complex).

A vector differentiation operator is defined as

\frac{d}{dx} = [\frac{∂}{∂x_{1}} , ⋯, \frac{∂}{∂x_{n}} ]^{T}  (A.262)

which, when applied to any scalar function f(x), produces its derivative with respect to x, the gradient vector of f(x):

g_{f}(x) = ▽f(x) = \frac{d}{dx}f(x) = [\frac{∂f}{∂x_{1}} , ⋯, \frac{∂f}{∂x_{n}} ]^{T} = [f_{x_{1}}, ⋯, f_{x_{n}}]^{T} (A.263)

The Directional derivative of a function f(x) is defined as the rate of change of the function value along a unit direction r = [r_{1}, ⋯, r_{n}]^{T} (with ||r|| = 1):

D_{r}f(x) = ▽_{r}f(x) = \frac{d}{dh}f(x + hr) = lim_{h→0} \frac{f(x + hr) − f(x)}{h}  (A.264)

Introducing n intermediate functions x_{i} +hr_{i} (i = 1, ⋯, n) of h, we can rewrite the above as the following by chain rule:

D_{r}f(x) = \frac{d}{dh}f(x+hr)|_{h=0} = ∑_{i=1}^{n} \frac{∂f(x + hr)}{∂(x_{i} + hr_{i})} \frac{d(x_{i} + hr_{i})}{dh}|_{h=0} = ∑_{i=1}^{n} f_{x_{i}} r_{i} = g_{f}^{T} r

(A.265)

We see that the directional derivative is the projection of the gradient vector onto the unit direction r, which is maximized when the gradient vector g_{f} is in the same direction as r. In other words, the function value increases mot rapidly along the direction of its gradient.

Vector differentiation has the following properties:

\frac{d}{dx}(a^{T} x) = \frac{d}{dx}(x^{T} a) = a,  \frac{d}{dx}(x^{T} x) = 2x,  \frac{d}{dx}(x^{T} Ax) = (A^{T} + A)x

(A.266)

To prove the third one, consider the kth element of the vector:

\frac{∂}{∂x_{k}} (x^{T} Ax) = \frac{∂}{∂x_{k}} ∑_{i=1}^{n} ∑_{j=1}^{n} a_{ij}x_{i}x_{j} = ∑_{i=1}^{n} a_{ik}x_{i} + ∑_{j=1}^{n} a_{kj}x_{j} (k = 1, ⋯, n)

(A.267)

Putting all n elements in vector form, we have the above. If A^{T} = A is symmetric, then we have

\frac{d}{dx}(x^{T} Ax) = 2Ax  (A.268)

In particular, when A = I, we have

\frac{d}{dx}(x^{T} x) = 2x  (A.269)

One can compare these results with the familiar derivatives in scalar case:

\frac{d}{dx}(ax^{2}) = 2ax  (A.270)

A matrix differentiation operator is defined as

\frac{d}{dA} = [ \begin{matrix}\frac{∂}{∂a_{11}} & ⋯ & \frac{∂}{∂a_{1n}} \\ ⋮ & ... & ⋮ \\ \frac{∂}{∂a_{m1}} & ⋯ & \frac{∂}{∂a_{mn}} \end{matrix} ]  (A.271)

which can be applied to any scalar function f(A):

\frac{d}{dA}f(A) = [ \begin{matrix}\frac{∂f(A)}{∂a_{11}} & ⋯ & \frac{∂f(A)}{∂a_{1n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f(A)}{∂a_{m1}} & ⋯ & \frac{∂f(A)}{∂a_{mn}} \end{matrix} ]  (A.272)

Specifically, consider f(A) = u^{T} Av, where u and v are m × 1 and n × 1 constant vectors, respectively, and A is an m × n matrix. Then we have:

\frac{d}{dA}(u^{T} Av) = uv^{T}  (A.273)

For more details refer to Matrix Cookbook at: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf.

A.5 The Fundamental Theorem of Linear Algebra

A.5.1 Rank-Nullity Theorem

Definition: The number of independent rows in a matrix is called the row rank, the number of independent columns in a matrix is called the column rank.

Theorem: The row rank and column rank of any matrix are the same.

Proof: Let the column rank of the following m × n matrix A be r

A = [a_{1}, ⋯, a_{r}, a_{r+1}, ⋯, a_{n}] = [A_{1}, A_{0}]  (A.274)

where A_{1} = [a_{1}, ⋯, a_{r}] is an m × r sub-matrix formed by the r linearly independent columns in A, assumed to be the first r columns (without loss of generality). Now consider the linear combination of another set of r n-D vectors {A_{1}^{T} a_{1}, ⋯, A_{1}^{T} a_{r}}:

∑_{i=1}^{r} c_{i}A_{1}^{T} a_{i} = A_{1}^{T} ∑_{i=1}^{r} c_{i}a_{i} = A_{1}^{T} A_{1}c = 0  (A.275)

which is assumed to be zero. As the r × r coefficient matrix A_{1}^{T} A_{1} has full rank r (A_{1} is formed by r independent vectors), we must have c = [c_{1}, ⋯, c_{r}]^{T} = 0, indicating {A_{1}^{T} a_{1}, ⋯, A_{1}^{T} a_{r}} are independent. However, as the linear combinations of r of the rows of A, these r independent vector A_{1}^{T} a_{i} (i = 1, ⋯, r) are in the row space of A. We therefore know that the row rank of A is no smaller than its column rank.

As the same argument can also be made for A^{T}, i.e., the column rank of A is no smaller than its row rank, we conclude that the column and row ranks of A must be the same. QED

Rank-nullity theorem: The sum of the rank of an m × n matrix A and the dimension of its nullspace N(A), the nullity of A, is n:

rank(A) + dim(N(A)) = rank(A) + Nullity(A) = n  (A.276)

Proof:

Let A = [a_{1}, ⋯, a_{r}, a_{r+1}, ⋯, a_{n}] be an m × n matrix of rank r. We assume, without loss of generality, the first r columns are independent, and the last n − r columns are dependent, and write it as A = [A_{1}, A_{0}]. Here

A_{1} = [a_{1}, ⋯, a_{r}]_{m×r}  (A.277)

is formed by the r independent columns of A, i.e., if A_{1}x = 0, then x = 0; and

A_{0} = [a_{r+1}, ⋯, a_{n}]_{m×(n−r)}  (A.278)

is formed by the n − r dependent columns of A, i.e., any of them can be written as a linear combination of those independent columns a_{1}, ⋯, a_{r} in A_{1}:

a_{r+k} = ∑_{i=1}^{r} c_{ik}a_{i} = [a_{1}, ⋯, a_{r}] [ c_{1k} ; ⋮ ; c_{rk} ] = A_{1}c_{k}, (k = 1, ⋯, n − r) (A.279)

where c_{k} = [c_{1k}, ⋯, c_{rk}]^{T} is a coefficient vector. We can put all these n − r equations together to write them in matrix form as

A_{0} = [a_{r+1}, ⋯, a_{n}] = A_{1}[c_{1}, ⋯, c_{n−r}] = A_{1}C  (A.280)

where C = [c_{1}, ⋯, c_{n−r}] is an r×(n−r) matrix composed of the n−r coefficient vectors. The matrix A can now be written as

A = [A_{1}, A_{0}] = [A_{1}, A_{1}C]  (A.281)

Define an n × (n − r) matrix

X = [ \begin{matrix}−C \\ I \end{matrix} ] = [ \begin{matrix}−c_{1} & ⋯ & −c_{n−r} \\ e_{1} & ⋯ & e_{n} \end{matrix} ] = [ \begin{matrix}−c_{1} & ⋯ & −c_{n−r} \\ 1 & ⋯ & 0 \\ 0 & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 0 \\ 0 & ⋯ & 1 \end{matrix} ] (A.282)

where I is an (n − r) × (n − r) identity matrix composed of n standard basis vectors e_{1}, ⋯, e_{n}. Now we have

AX = [A_{1}, A_{1}C] [ \begin{matrix}−C \\ I \end{matrix} ] = −A_{1}C + A_{1}C = O  (A.283)

indicating any column x in X is a solution of the homogeneous equation Ax = 0, i.e., x is in the nullspace N(A). Also, if we let Xu = 0, we get

Xu = [ \begin{matrix}−C \\ I \end{matrix} ] u = [ \begin{matrix}−Cu \\ u \end{matrix} ] = [ \begin{matrix}0 \\ 0 \end{matrix} ] = 0  (A.284)

i.e., u = 0, indicating all columns in X are independent.

On the other hand, we can also show that any solution of the homogeneous equation Ax = 0 must be a linear combination of the columns of X. Assuming x is such a solution, i.e.,

Ax = 0 = [A_{1}, A_{1}C] [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = A_{1}(x_{1} + Cx_{2})  (A.285)

As all columns in A are independent, we must have

x_{1} + Cx_{2} = 0, or x_{1} = −Cx_{2}  (A.286)

and we get

x = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−Cx_{2} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−C \\ I \end{matrix} ] x_{2} = Xx_{2}  (A.287)

indicating x is a linear combination of the n − r columns in X.

Based on the two results obtained above, we can now conclude that the nullspace N(A) is spanned by the n − r independent column vectors in X, i.e., the dimension of N(A) is n − r, and rank(A) + dim(A) = n.

A.5.2 Solving Linear Equation Systems

When solving a linear equation system Ax = b with an M ×N coefficient matrix A, we need to answer some questions such as the following:

• Does a solution exist, i.e., can we find an x^{∗} so that Ax^{∗} = b holds?

• If such a solution exists, is it unique? If it is not unique, how can we find all of the solutions?

• If no solution exists, can we still find the optimal approximated solution x̂ so that the error ||e|| = ||Ax̂ − b|| is minimized?

• If the system has fewer equations than unknowns (M < N), are there infinite solutions?

• If the system has more equations than unknowns (M > N), is there no solution?

To answer all such questions, we consider the fundamental theorem of linear algebra to reveal the structure of the solutions of a given linear system Ax = b.

The M × N coefficient matrix A ∈ R^{M×N} can be expressed in terms of either its N M-D column vectors c_{j} or its M N-D row vectors r_{i}^{T}:

A = [ \begin{matrix}a_{11} & ⋯ & a_{1N} \\ ⋮ & ... & ⋮ \\ a_{M1} & ⋯ & a_{MN} \end{matrix} ] = [c_{1}, ⋯, c_{N}] = [ r_{1}^{T} ; ⋮ ; r_{M}^{T} ] ,  (A.288)

A^{T} = [ \begin{matrix}a_{11} & ⋯ & a_{M1} \\ ⋮ & ... & ⋮ \\ a_{1N} & ⋯ & a_{MN} \end{matrix} ] = [r_{1}, ⋯, r_{M}] = [ c_{1}^{T} ; ⋮ ; c_{N}^{T} ]  (A.289)

where the ith row r_{i}, (i = 1, ⋯, M) and the jth column c_{j}, (j = 1, ⋯, N) of A are both represented as column vectors:

c_{j} = [ a_{1j} ; ⋮ ; a_{Mj} ] ,  r_{i} = [ a_{i1} ; ⋮ ; a_{iN} ] i.e. r_{i}^{T} = [a_{i1}, ⋯, a_{iN}]  (A.290)

In general a function y = f(x) can be represented by f : X → Y, where

• X is the domain of the function, the set of all input or argument values;

• Y is the codomain of the function, the set into which all outputs of the function fall;

• The image of f(x) is the set formed by the outputs of the function corresponding to all possible x ∈ X, a subset of the codomain.

A linear function f(x) = Ax maps an N-D vector x ∈ ℝ^{N} in the domain of the function into an M-D vector Ax ∈ R^{M} in the codomain of the function. The fundamental theorem of linear algebra concerns the following four subspaces all associated with with the M × N coefficient matrix A with rank R = rank(A) ≤ min(M, N), i.e., A has R independent columns and rows.

• The column space of A is a space spanned by its N M-D column vectors (of which R ≤ N are independent):

C(A) = span(c_{1}, ⋯, c_{N}) ⊆ R^{M}  (A.291)

This is an R-D subspace of R^{M} composed of all possible linear combinations of its N column vectors:

x_{1}c_{1} + ⋯ + x_{N}c_{N} = [c_{1}, ⋯, c_{N}] [ x_{1} ; ⋮ ; x_{N} ] = Ax (for any x ∈ ℝ^{N})

(A.292)

The column space C(A) is the image of the linear transformation f(x) = Ax, and the equation Ax = b is solvable if and only if b ∈ C(A). The dimension of the column space is the rank of A, dim C(A) = rank(A) = R.

• The row space of A is a space spanned by its M N-D row vectors (of which R ≤ M are independent):

R(A) = span(r_{1}, ⋯, r_{M}) ⊆ ℝ^{N}  (A.293)

This is an R-D subspace of ℝ^{N} composed of all possible linear combinations of its M row vectors:

y_{1}r_{1} + ⋯ + y_{M}r_{N} = [r_{1}, ⋯, r_{M}] [ y_{1} ; ⋮ ; y_{M} ] = A^{T} y (for any y ∈ ℝ^{M})

(A.294)

The row space R(A) is the image of the linear transformation f(y) = A^{T} y, and the equation A^{T} y = c is solvable if and only if c ∈ R(A). As the rows and columns in A are respectively the columns and rows in A^{T}, the row space of A is the column space of A^{T}, and the column space of A is the row space of A^{T}:

R(A) = C(A^{T}),  C(A) = R(A^{T})  (A.295)

The rank R = rank(A) is the number of linearly independent rows and columns of A, i.e., the row space and the column space have the same dimension, both equal to the rank of A:

dim C(A) = dim R(A) = rank(A) = R  (A.296)

• The null space (kernel) of A, denoted by N(A), is the set of all N-D vectors x that satisfy the homogeneous equation

Ax = [ r_{1}^{T} ; ⋮ ; r_{M}^{T} ] x = 0, or r_{i}^{T} x = 0 (i = 1, ⋯, M)  (A.297)

i.e.,

N(A) = {x ∈ ℝ^{N}|Ax = 0} ⊆ ℝ^{N}  (A.298)

In particular, when x = 0, we get Ax = 0, i.e., the origin 0 ∈ N(A) is in the null space.

As r_{i}^{T} x = 0 for any x ∈ N(A) and r_{i} ∈ R(A), we see that the null space N(A) and the row space R(A) = C(A^{T}) are orthogonal to each other, N(A) ⊥ R(A).

The dimension of the null space is called the nullity of A: dim N(A) = nullity(A). The rank-nullity theorem states the sum of the rank and the nullity of an M × N matrix A is equal to N:

rank(A) + nullity(A) = dim R(A) + dim N(A) = N  (A.299)

We therefore see that R(A) and N(A) are two mutually exclusive and complementary subspaces of ℝ^{N}:

R(A) ⊥ N(A), R(A) ∩ N(A) = ∅, R(A) ⊕ N(A) = ℝ^{N} (A.300)

i.e., they are orthogonal complement of each other, denoted by

N(A) = R(A)^{⊥}, R(A) = N(A)^{⊥}  (A.301)

Any N-D vector x ∈ ℝ^{N} is in either of the two subspaces R(A) and N(A).

• The null space of A^{T} (left null space of A), denoted by N(A^{T}), is the set of all M-D vectors y that satisfy the homogeneous equation

A^{T} y = [c_{1}, ⋯, c_{N}]^{T} y = [ c_{1}^{T} ; ⋮ ; c_{N}^{T} ] y = 0, or c_{j}^{T} y = 0, (j = 1, ⋯, N)

(A.302)

i.e.,

N(A^{T}) = {y ∈ ℝ^{M}|A^{T} y = 0} ⊆ ℝ^{M}  (A.303)

As all y ∈ N(A^{T}) are orthogonal to c_{i} ∈ C(A), N(A^{T}) is orthogonal to the column space C(A) = R(A^{T}):

N(A^{T}) ⊥ C(A),  N(A^{T}) ⊥ R(A^{T})  (A.304)

We see that C(A) and N(A^{T}) are two mutually exclusive and complementary subspaces of ℝ^{M}:

C(A) ⊥ N(A^{T}), C(A) ∩ N(A^{T}) = ∅, C(A) ⊕ N(A^{T}) = R^{M},

i.e. N(A^{T}) = C(A)^{⊥}, C(A) = N(A^{T})^{⊥}  (A.305)

Any M-D vector y ∈ ℝ^{M} is in either of the two subspaces C(A) and N(A^{T}).

The four subspaces are summarized in the figure below, showing the domain ℝ^{N} = R(A) ⊕ N(A) (left) and the codomain ℝ^{M} = C(A) ⊕ N(A^{T}) (right) of the linear mapping Ax = b, where

• x_{p} ∈ R(A) is the particular solution that is mapped to Ax_{p} = b ∈ C(A), the image of f(x) = Ax;

• x_{h} ∈ N(A) is a homogeneous solution that is mapped to Ax_{h} = 0 ∈ ℝ^{M};

• x_{c} = x_{p} + x_{h} is the complete solution that is mapped to Ax_{c} = b ∈ C(A).

On the other hand, y_{p} ∈ C(A) = R(A^{T}), y_{h} ∈ N(A^{T}), and y_{c} = y_{p} + y_{h} are respectively the particular, homogeneous and complete solutions of A^{T} y = c. Here we have assumed b ∈ C(A) and c ∈ R(A), i.e., both Ax = b and A^{T} y = c are solvable. We will also consider the case where b ∉ C(A) later.

We now consider specifically how to find the solutions of the system Ax = b in light of the four subspaces of A defined above, through the examples below.

Example A.4 Solve the homogeneous equation system:

Ax = [ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 0 \end{matrix} ] = 0  (A.306)

We first convert A into the rref:

[ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 \\ 0 & −1 & −2 \\ 0 & −5 & −10 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ]

→ [ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}0 \\ 0 \\ 0 \end{matrix} ]  (A.307)

The R = rank(A) = 2 columns in the rref containing a single 1, called a pivot, are called the pivot columns, and the rows containing a pivot are called the pivot rows. Here, R = 2 < M = N = 3, i.e., A is a singular matrix. The two pivot rows r_{1}^{T} = [1, 0, 1] and r_{2}^{T} = [0, 1, 2] can be used as the basis vectors that span the row space R(A):

R(A) = c_{1}r_{1} + c_{2}r_{2} = c_{1} [ \begin{matrix}1 \\ 0 \\ 1 \end{matrix} ] + c_{2} [ \begin{matrix}0 \\ 1 \\ 2 \end{matrix} ]  (A.308)

Note that the pivot columns of the rref do not span the column space C(A), as the row reduction operations do not reserve the columns of A. But they indicate the corresponding columns c_{1} = [1 2 3]^{T} and c_{2} = [2 3 1]^{T} in the original matrix A can be used as the basis that spans C(A). In general the bases of the row and column spaces so obtained are not orthogonal.

The R pivot rows are the independent equations in the system of M equations, and the variables corresponding to the pivot columns (here x_{1} and x_{2}) are the pivot variables. The remaining M − R non-pivot rows containing all zeros are not independent, and the variables corresponding to the non-pivot rows are free variables (here x_{3}), which can take any values.

From the rref form of the equation, we get

x_{1} + x_{3} = 0, x_{2} + 2x_{3} = 0  (A.309)

If we let the free variable x_{3} take the value 1, then we can get the two pivot variables x_{1} = −1 and x_{2} = −2, and a special homogeneous solution x_{h} = [−1 − 2 1]^{T} as a basis vector that spans the 1-D null space N(A). However, as the free variable x_{3} can take any value c, the complete solution is the entire 1-D null space:

N(A) = cx_{h} = c [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ]  (A.310)

Example A.5 Solve the non-homogeneous equation with the same coefficient matrix A used in the previous example:

Ax = [ \begin{matrix}1 & 2 & 5 \\ 2 & 3 & 8 \\ 3 & 1 & 5 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}19 \\ 31 \\ 22 \end{matrix} ] = b  (A.311)

We use Gauss-Jordan elimination to solve this system:

[ A b ] = [ \begin{matrix}1 & 2 & 5 & 19 \\ 2 & 3 & 8 & 31 \\ 3 & 1 & 5 & 22 \end{matrix} ] → [ \begin{matrix}1 & 2 & 5 & 19 \\ 0 & −1 & −2 & −7 \\ 0 & −5 & −10 & −35 \end{matrix} ]

→ [ \begin{matrix}1 & 2 & 5 & 19 \\ 0 & 1 & 2 & 7 \\ 0 & 0 & 0 & 0 \end{matrix} ] → [ \begin{matrix}1 & 0 & 1 & 5 \\ 0 & 1 & 2 & 7 \\ 0 & 0 & 0 & 0 \end{matrix} ]  (A.312)

The R = 2 pivot rows correspond to the independent equations in the system, i.e., rand(A) = 2, while the remaining M − R = 1 non-pivot row does not play any role as they map any x to 0. As A is singular, A^{−1} does not exist. However, we can find the solution based on the rref of the system, which can also be expressed in block matrix form:

[ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ]  or  [ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}x_{pivot} \\ x_{free} \end{matrix} ] = [ \begin{matrix}b_{1} \\ 0 \end{matrix} ]

(A.313)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}1 \\ 2 \end{matrix} ] , x_{pivot} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , x_{free} = x_{3}, b_{1} = [ \begin{matrix}5 \\ 7 \end{matrix} ]

(A.314)

Solving the matrix equation above for x_{pivot}, we get

Ix_{pivot}+Fx_{free} = b_{1}, x_{pivot} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = b_{1}−Fx_{free} = [ \begin{matrix}5 \\ 7 \end{matrix} ]+x_{3} [ \begin{matrix}−1 \\ −2 \end{matrix} ]

(A.315)

If we let x_{free} = x_{3} = 0, we get a particular solution x_{p} = [5 7 0]^{T}, which can be expressed as a linear combination of r_{1} and r_{2} that span R(A), and x_{h} that span N(A):

x_{p} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] = \frac{11}{6}r_{1}+ \frac{2}{3}r_{2}− \frac{19}{6}x_{h} = \frac{11}{6} [ \begin{matrix}1 \\ 0 \\ 1 \end{matrix} ]+ \frac{2}{3} [ \begin{matrix}0 \\ 1 \\ 2 \end{matrix} ]− \frac{19}{6} [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ] (A.316)

We see that this solution is not entirely in the row space R(A). In general, this is the case for all particular solutions so obtained.

Having found both the particular solution x_{p} and the homogeneous solution x_{h}, we can further find the complete solution x_{c} as the sum of x_{p} and the entire null space spanned by x_{h}:

x_{c} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] + c [ \begin{matrix}−1 \\ −2 \\ 1 \end{matrix} ] = x_{p} + cx_{h} = x_{p} + N(A)  (A.317)

Based on different constant c, we get a set of equally valid solutions. For example, if c = 0, 1, 2, then we get

x_{0} = [ \begin{matrix}5 \\ 7 \\ 0 \end{matrix} ] , x_{1} = [ \begin{matrix}4 \\ 5 \\ 1 \end{matrix} ] , x_{2} = [ \begin{matrix}3 \\ 3 \\ 2 \end{matrix} ]  (A.318)

These solutions have the same projection onto the row space R(A), i.e., they have the same projections onto the two basis vectors r_{1} and r_{2} that span R(A):

x_{0}^{T} r_{1} = x_{1}^{T} r_{1} = x_{2}^{T} r_{1} = 5,  x_{0}^{T} r_{2} = x_{1}^{T} r_{2} = x_{2}^{T} r_{2} = 7  (A.319)

The figure below shows how the complete solution x_{c} can be obtained as the sum of a particular solution x_{p} in R(A) and the entire null space N(A). Here N = 3 and space ℝ^{3} is composed of R(A) and N(A), respectively 2-D and 1-D on the left, but 1-D and 2-D on the right. In either case, the complete solution is any particular solution plus the entire null space, the vertical dashed line on the left, the top dashed plane on the right. All points on the vertical line or top satisfy the equation system, as they are have the same projection onto the row space R(A).

If the right hand side is b′ = [19, 31, 20]^{T}, then the rref of the equation becomes:

[ \begin{matrix}1 & 0 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \end{matrix} ] = [ \begin{matrix}5 \\ 7 \\ 2 \end{matrix} ]  (A.320)

The non-pivot row is an impossible equation 0 = 2, indicating that no solution exists, as b′ ∉ C(A) is not in the column space spanned by r_{1} = [1 0 1]^{T} and r_{2} = [0 1 2]^{T}.

Example A.6 Find the complete solution of the following linear equation system:

Ax = [ \begin{matrix}1 & 2 & 3 & 4 \\ 4 & 3 & 2 & 1 \\ −2 & 1 & 4 & 7 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = b  (A.321)

This equation can be solved in the following steps:

• Construct the augmented matrix and then convert it to the rref form:

[ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 4 & 3 & 2 & 1 & 2 \\ −2 & 1 & 4 & 7 & 4 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & −5 & −10 & −15 & −10 \\ 0 & 5 & 10 & 15 & 10 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & 1 & 2 & 3 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ]

→ [ \begin{matrix}1 & 0 & −1 & −2 & −1 \\ 0 & 1 & 2 & 3 & 2 \\ 0 & 0 & 0 & 0 & 0 \end{matrix} ] = [ \begin{matrix}I & F & b_{1} \\ 0 & 0 & 0 \end{matrix} ]

The two pivot rows r_{1}^{T} = [1 0 − 1 − 2] and r_{2}^{T} = [0 1 2 3] in the rref span R(A), and the two columns in the original matrix A corresponding to the pivot columns in the rref, [1 4 − 2]^{T} and [2 3 1]^{T} could be used as the basis vectors that span C(A).

The equation system can be represented in block matrix form:

[ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}x_{p} \\ x_{f} \end{matrix} ] = [ \begin{matrix}b_{1} \\ 0 \end{matrix} ]  (A.322)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}−1 & −2 \\ 2 & 3 \end{matrix} ] , x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , x_{f} = [ \begin{matrix}x_{3} \\ x_{4} \end{matrix} ] , b_{1} = [ \begin{matrix}−1 \\ 2 \end{matrix} ]

(A.323)

Multiplying out we get

x_{p} + Fx_{f} = b_{1}  (A.324)

• Find the homogeneous solution for equation Ax = 0 by setting b_{1} = 0:

Ix_{p} + Fx_{f} = 0, i.e. x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = −Fx_{f} = [ \begin{matrix}1 & 2 \\ −2 & −3 \end{matrix} ] [ \begin{matrix}x_{3} \\ x_{4} \end{matrix} ]

(A.325)

We let x_{f} be either of the two standard basis vectors x_{f} = [x_{3}, x_{4}]^{T} = [1 0]^{T} and x_{f} = [x_{3}, x_{4}]^{T} = [0 1]^{T} of the null space N(A), and get

x_{p} = [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}1 \\ −2 \end{matrix} ] or [ \begin{matrix}2 \\ −3 \end{matrix} ]  (A.326)

and the two corresponding homogeneous solutions:

x_{h1} = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] ,  x_{h2} = [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ] (A.327)

• Find the particular solution of the non-homogeneous equation Ax = b by setting x_{f} = 0

Ix_{p} + Fx_{f} = x_{p} = b_{1}, i.e. [ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] = [ \begin{matrix}−1 \\ 2 \end{matrix} ] i.e. x_{p} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ]

(A.328)

• Find the complete solution:

x_{c} = x_{p} +N(A) = x_{p} +c_{1}x_{h1} +c_{2}x_{h2} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ]+c_{1} [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ]+c_{2} [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ]

(A.329)

If the right-hand side is b′ = [1, 3, 5]^{T}, then the row reduction of the augmented matrix yields:

[ \begin{matrix}1 & 2 & 3 & 4 & 1 \\ 4 & 3 & 2 & 1 & 3 \\ −2 & 1 & 4 & 7 & 5 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 3 \\ 0 & −5 & −10 & −15 & −1 \\ 0 & 5 & 10 & 15 & 7 \end{matrix} ] → [ \begin{matrix}1 & 2 & 3 & 4 & 1 \\ 0 & 1 & 2 & 3 & 0.2 \\ 0 & 0 & 0 & 0 & 1.2 \end{matrix} ]

(A.330)

The equation corresponding to the last non-pivot row is 0 = 1.2, indicating the system is not solvable (even though the coefficient matrix does not have full rank), because b′ ∉ C(A) is not in the column space.

Example A.7 Consider the linear equation system with a coefficient matrix A^{T}, the transpose of A used in the previous example:

A^{T} y = [ \begin{matrix}1 & 4 & −2 \\ 2 & 3 & 1 \\ 3 & 2 & 4 \\ 4 & 1 & 7 \end{matrix} ] [ \begin{matrix}y_{1} \\ y_{2} \\ y_{3} \end{matrix} ] = [ \begin{matrix}3 \\ 11 \\ 19 \\ 27 \end{matrix} ] = c  (A.331)

• Convert the augmented matrix into the rref form:

[ \begin{matrix}1 & 4 & −2 & 3 \\ 2 & 3 & 1 & 11 \\ 3 & 2 & 4 & 19 \\ 4 & 1 & 7 & 27 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 3 \\ 0 & −5 & 5 & 5 \\ 0 & −10 & 10 & 10 \\ 0 & −15 & 15 & 15 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 3 \\ 0 & 1 & −1 & −1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ] → [ \begin{matrix}1 & 0 & 2 & 7 \\ 0 & 1 & −1 & −1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ]

(A.332)

The two pivot rows r_{1}^{T} = [1 0 2] and r_{2}^{T} = [0 1 −1] are the basis vectors that span R(A^{T}) = C(A). The two vectors that span C(A) found in Example A.6 can be expressed as linear combinations of r_{1} and r_{2}, [1 4 − 2]^{T} = r_{1} + 4r_{2} and [2 3 1]^{T} = 2r_{1} + 3r_{2}, i.e., either of the two pairs can be used as the basis that span C(A).

Based on the rref above, the equation system can now be written as

[ \begin{matrix}I & F \\ 0 & 0 \end{matrix} ] [ \begin{matrix}y_{p} \\ y_{f} \end{matrix} ] = [ \begin{matrix}c_{1} \\ 0 \end{matrix} ]  (A.333)

where

I = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] , F = [ \begin{matrix}2 \\ −1 \end{matrix} ] , y_{p} = [ \begin{matrix}y_{1} \\ y_{2} \end{matrix} ] , y_{f} = y_{3}, c_{1} = [ \begin{matrix}7 \\ −1 \end{matrix} ]

(A.334)

Multiplying out we get

Iy_{p} + Fy_{f} = c_{1}  (A.335)

• Find the homogeneous solution for A^{T} y = c = 0 by setting c = 0 and thereby c_{1} = 0. We let the free variable y_{f} = y_{3} = 1 and get

Iy_{p} + Fy_{f} = 0, i.e. y_{p} = −Fy_{f} = [ \begin{matrix}y_{1} \\ y_{2} \end{matrix} ] = − [ \begin{matrix}2 \\ −1 \end{matrix} ] y_{3} = [ \begin{matrix}−2 \\ 1 \end{matrix} ] ,

i.e.,  y_{h} = [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]  (A.336)

• Find the particular solution of the non-homogeneous equation A^{T} y = c ≠ 0 by setting y_{f} = y_{3} = 0:

Iy_{p} + Fy_{f} = y_{p} = c_{1} = [ \begin{matrix}7 \\ −1 \end{matrix} ] , i.e. y_{p} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ]  (A.337)

• Find the complete solution:

y_{c} = y_{p} + N(A^{T}) = y_{p} + cy_{3} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ] + c [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]  (A.338)

If the right-hand side is c′ = [1, 1, 2, 3]^{T},

[ \begin{matrix}1 & 4 & −2 & 1 \\ 2 & 3 & 1 & 1 \\ 3 & 2 & 4 & 2 \\ 4 & 1 & 7 & 3 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 1 \\ 0 & −5 & 5 & −1 \\ 0 & −10 & 10 & −1 \\ 0 & −15 & 15 & −1 \end{matrix} ] → [ \begin{matrix}1 & 4 & −2 & 1 \\ 0 & 1 & −1 & 1/5 \\ 0 & 0 & 0 & 1 \\ 0 & 0 & 0 & 2 \end{matrix} ]

(A.339)

indicating the system is not solvable, as this c′ ∉ C(A^{T}) = R(A), i.e., it is not in the column space of A^{T} or row space of A.

In these examples, we have obtained all four subspaces associated with this matrix A with M = 3, N = 4, and R = 2, in terms of the bases that span the subspaces:

1. The row space R(A) = C(A^{T}) is an R-D subspace of ℝ^{4}, spanned by the R = 2 pivot rows of the rref of A:

R(A) = C(A^{T}) = span ([ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ] , [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ])  (A.340)

2. The null space N(A) is an (N-R)-D subspace of ℝ^{4} spanned by the N − R independent homogeneous solutions:

N(A) = span ([ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] , [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ])  (A.341)

Note that the basis vectors of N(A) are indeed orthogonal to those of R(A).

3. The column space of A is the same as the row space of A^{T}, which is the R-D subspace of R^{3} spanned by the two pivot rows of the rref of A^{T}.

C(A) = R(A^{T}) = span ([ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ] , [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ])  (A.342)

4. The left null space N(A^{T}) is a (M-R)-D subspace of ℝ^{3} spanned by the homogeneous solutions (here one 3 − 2 = 1 solution):

N(A^{T}) = span ([ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]) = c [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ]  (A.343)

Again note that the basis vectors of N(A^{T}) are orthogonal to those of C(A).

In general, here are the ways to find the bases of the four subspaces:

• The basis vectors of R(A) are the pivot rows of the rref of A.

• The basis vectors of C(A) are the pivot rows of the rref of A^{T}.

• The basis vectors of N(A) are the independent homogeneous solutions of Ax = 0. To find them, reduce A to the rref, identify all free variables x_{f} corresponding to non-pivot columns, set one of them to 1 and the rest to 0, solve homogeneous system Ax = 0 to find the pivot variables x_{p} to get one basis vector. Repeat the process for each of the free variables to get all basis vectors.

• The basis vectors of R(A^{T}) can be obtained by doing the same as above for A^{T}.

Note that while the basis of R(A) are the pivot rows of the rref of A, as its rows are equivalent to those of A, the pivot columns of the rref basis of C(A) are not the basis of C(A), as the columns of A have been changed by the row deduction operations and are therefore not equivalent to the columns of the resulting rref. The columns in A corresponding to the pivot columns in the rref could be used as the basis of C(A). Alternatively, the basis of C(A) can be obtained from the rref of A^{T}, as its rows are equivalent to those of A^{T}, which are the columns of A.

We further make the following observations:

• The basis vectors of each of the four subspaces are independent, the basis vectors of R(A) and N(A) are orthogonal, and dim R(A) + dim N(A) = M. Similarly, the basis vectors of C(A) and N(A^{T}) are orthogonal, and dim C(A) + dim N(A^{T}) = N. In other words, the four subspaces indeed satisfy the following orthogonal and complementary properties:

C(A) ⊥ N(A^{T}), R(A) ⊥ N(A), C(A)⊕N(A^{T}) = R^{3}, R(A)⊕N(A) = R^{4}

(A.344)

i.e., they are orthogonal complements: C(A) = N(A)^{⊥}, R(A) = N(A)^{⊥}.

• For Ax = b to be solvable, the constant vector b on the right-hand side must be in the column space, b ∈ C(A). Otherwise the equation is not solvable, even if R < M < N. Similarly, for A^{T} x = c to be solvable, c must be in the row space c ∈ R(A). In the examples above, both b and c are indeed in their corresponding column spaces:

b = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = 3 [ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ] + 2 [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ] ∈ C(A)

c = [ \begin{matrix}3 \\ 11 \\ 19 \\ 27 \end{matrix} ] = 3 [ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ] + 11 [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ] ∈ R(A)  (A.345)

But as b = [1, 3, 5]^{T} ∉ C(A) and c = [1, 1, 2, 3]^{T} ∉ R(A), the corresponding systems have no solutions.

• All homogeneous solutions of Ax = 0 are in the null space x_{h} ∈ N(A), but in general the particular solutions x_{p} ∈ R^{N} = R(A) ⊕ N(A) are not necessarily in the row space R(A). In the examples above, x_{p} ∈ R^{N} is a linear combination of the R = 2 basis vectors of R(A) and N −R = 2 basis vectors of N(A):

x_{p} = [ \begin{matrix}−1 \\ 2 \\ 0 \\ 0 \end{matrix} ] = (0.1 [ \begin{matrix}1 \\ 0 \\ −1 \\ −2 \end{matrix} ] + 0.2 [ \begin{matrix}0 \\ 1 \\ 2 \\ 3 \end{matrix} ])+(−0.3 [ \begin{matrix}1 \\ −2 \\ 1 \\ 0 \end{matrix} ] − 0.4 [ \begin{matrix}2 \\ −3 \\ 0 \\ 1 \end{matrix} ]) = x′_{p}+x′′_{p}

where

x′_{p} = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ] ∈ R(A), x′′_{p} = [ \begin{matrix}−1.1 \\ 1.8 \\ −0.3 \\ −0.4 \end{matrix} ] ∈ N(A)  (A.347)

are the projections of x_{p} onto R(A) and N(A), respectively, and x′_{p} is another particular solution without any homogeneous component that satisfies Ax = b, while x′′_{p} is a homogeneous solution satisfying Ax = 0.

• All homogeneous solutions of A^{T} y = c are in the left null space y_{h} ∈ N(A^{T}), but in general the particular solutions y_{p} ∈ C(A) ⊕ N(A^{T}) = R^{M} are not necessarily in the column space. In the examples above, y_{p} ∈ R^{M} is a linear combination of the R = 2 basis vectors of C(A) and M − R = 1 basis vector of N(A^{T}):

y_{p} = [ \begin{matrix}7 \\ −1 \\ 0 \end{matrix} ] = (2 [ \begin{matrix}1 \\ 0 \\ 2 \end{matrix} ] + 1.5 [ \begin{matrix}0 \\ 1 \\ −1 \end{matrix} ])−2.5 [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ] = [ \begin{matrix}2 \\ 1.5 \\ 2.5 \end{matrix} ]−2.5 [ \begin{matrix}−2 \\ 1 \\ 1 \end{matrix} ] = y′_{p}+y′′_{p}

(A.348)

where y′_{p} = [2, 1.5, 2.5]^{T} ∈ C(A) is a particular solution (without any homogeneous component) that satisfies A^{T} y = c.

Here is a summary of the four subspaces associated with an M by N matrix A of rank R.

dimR(A) dimN(A) dimC(A) dimR(A^{T}) solvability of Ax = b
M = N = R  R  0  R  0  unique solution x = A^{−1}b
M > N = R  R  0  R  M − R  over-constrained, solvable if b ∈ C(A)
N > M = R  R  N − R  R  0  under-constrained, infinite solutions x = x_{p} + N(A)
R < min(M, N)  R  N − R  R  M − R  solvable if b ∈ C(A), infinite solutions

(A.349)

The figure below illustrates a specific case with M = N = 3 and R = 2. As b ∉ C(A), the system can only be approximately solved to find Ax = p ∈ C(A), which is the projection of b onto the column space C(A). The error ||e|| = ||p − b|| = ||Ax_{p} − b|| is minimized, x_{p} is the optimal approximation. We will consider ways to obtain this optimal approximation in the following sections.

A.6 Pseudo-Inverse

A.6.1 Pseudo-Inverse

The inverse of an m × n matrix A does not exist if it is not square m ≠ n. But we can still find its pseudo-inverse, an n × m matrix denoted by A^{−}, if rank(A) = r = min(m, n), in either of the following ways:

• If m > n = r, A^{T} A is an n × n full-rank invertible matrix, and we define the left inverse:

A^{−} = (A^{T} A)^{−1}A^{T}  (A.350)

which satisfies

A^{−}A = (A^{T} A)^{−1}A^{T} A = I_{n×n}  (A.351)

But

AA^{−} = A(A^{T} A)^{−1}A^{T} ≠ I  (A.352)

• If n > m = r, AA^{T} is an m × m full-rank invertible matrix, and we define the right inverse:

A^{−} = A^{T} (AA^{T})^{−1}  (A.353)

which satisfies

AA^{−} = AA^{T} (AA^{T})^{−1} = I_{m×m}  (A.354)

But

A^{−}A = A^{T} (AA^{T})^{−1}A ≠ I  (A.355)

The left and right inverses defined respectively for m > n and m < n are the same. If m > n, then we have the left inverse A^{−} = (A^{T} A)^{−1}A^{T}. We then define B = A^{T}, and take the transpose on A^{−} to get

(A^{−})^{T} = [(A^{T} A)^{−1}A^{T}]^{T} = A(A^{T} A)^{−1} = (A^{T})^{−};  (A.356)

i.e., B^{−} = B^{T} (BB^{T})^{−1}, which is the second definition of the pseudo inverse of B.

We can show that (A^{−})^{−} = A. If m > n, then we have

(A^{−})^{−} = [(A^{T} A)^{−1}A^{T}]^{−} = [(A^{T} A)^{−1}A^{T}]^{T} [(A^{T} A)^{−1}A^{T} [(A^{T} A)^{−1}A^{T}]^{T} ]^{−1}
= A(A^{T} A)^{−1} [(A^{T} A)^{−1}A^{T} A(A^{T} A)^{−1}]^{−1}
= A(A^{T} A)^{−1}(A^{T} A) = A.

Similarly we can show this is the case if m < n.

In particular, when m = n, A is a square invertible matrix, and both the left and right pseudo-inverse defined above become the regular inverse

A^{−} = (A^{T} A)^{−1}A^{T} = A^{−1}(A^{T})^{−1}A^{T} = A^{−1}
A^{−} = A^{T} (AA^{T})^{−1} = A^{T} (A^{T})^{−1}A^{−1} + A^{−1}

If r < min(m, n), then A^{−1} does not exist, and A^{−} is neither left nor right inverse, because A^{−}A ≠ I and AA^{−} ≠ I, as they are not full rank.

The pseudo inverse of A can also be obtained based on its SVD A_{m×n} = U_{m×m}Σ_{m×n}V_{n×n}^{T}. Consider the following two cases:

• If m > n, the left pseudo inverse is

A^{−} = (A^{T} A)^{−1}A^{T} = [(UΣV^{T})^{T} (UΣV^{T})]^{−1}(UΣV^{T})^{T}
= (VΣ^{T} U^{T} UΣV^{T})^{−1}(VΣ^{T} U^{T}) = V(Σ^{T} Σ)^{−1}V^{T} VΣ^{T} U^{T}
= V[(Σ^{T} Σ)^{−1}Σ^{T}]U^{T} = VΣ^{−}U^{T}

• If m < n, the right pseudo inverse is

A^{−} = A^{T} (AA^{T})^{−1} = (UΣV^{T})^{T} [(UΣV^{T})(UΣV^{T})^{T}]^{−1}
= (VΣ^{T} U^{T})(UΣV^{T} VΣ^{T} U^{T})^{−1} = VΣ^{T} U^{T} (UΣΣ^{T} U^{T})^{−1}
= VΣ^{T} U^{T} U(ΣΣ^{T})^{−1}U^{T} = V[Σ^{T} (ΣΣ^{T})^{−1}]U^{T} = VΣ^{−}U^{T}

In either case, we have

A_{n×m}^{−} = (U_{m×m}Σ_{m×m}V_{n×n}^{T})^{−} = V_{n×n}Σ_{n×m}^{−}U_{m×m}^{T}  (A.357)

where Σ_{n×m}^{−} is the pseudo inverse of Σ_{m×n} with 1/σ_{1}, ⋯, 1/σ_{r} on its diagonal. We also have

A^{−}A = (VΣ^{−}U^{T})(UΣV^{T}) = V(Σ^{−}Σ)V^{T}  (A.358)

if r = m, then Σ^{−}Σ = I an n×n identity matrix and we have A^{−}A = VV^{T} = I. Otherwise if r < m, Σ^{−}Σ has zero components on the diagonal, and A^{−}A ≠ I.

A.6.2 Pseudo-Inverse Solutions Based on SVD

In the previous section we obtained the solution of the equation Ax = b together with the bases of the four subspaces of A based its rref. Here we will consider an alternative and better way to solve the same equation and find a set of orthogonal bases that also span the four subspaces, based on the pseudo-inverse and the singular value decomposition (SVD) of A. The solution obtained this way is optimal in some certain sense as shown below.

Consider the SVD of an M × N matrix A of rank R ≤ min(M, N):

A = UΣV^{T}  (A.359)

where

Σ = [ \begin{matrix}σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{M×N}  (A.360)

is an M × N matrix with R non-zero singular values σ_{i} = \sqrt{λ_{i}} (i = 1, …, R) of A along the diagonal (starting from the top-left corner), while all other components are zero, and U = [u_{1}, ⋯, u_{M}] and V = [v_{1}, ⋯, v_{N}] are two orthogonal matrices of dimensions M × M and N × N respectively. The column vectors u_{i} (i = 1, ⋯, M) and v_{j} (j = 1, ⋯, N), are called the left and right singular vectors of A, respectively, and they can be used as the orthonormal bases to span respectively ℝ^{M} = C(A) ⊕ N(A^{T}) and its subspaces C(A) and N(A^{T}), and ℝ^{N} = C(A) ⊕ N(A) and its subspaces C(A) and N(A).

To see this, we rewrite the SVD of A as:

A = [c_{1}, ⋯, c_{N}] = UΣV^{T}

= [u_{1}, · · · · · · · · · , u_{M}] [ \begin{matrix}σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{M×N} [ v_{1}^{T} ; ⋮ ; ⋮ ; ⋮ ; v_{N}^{T} ]

= ∑_{k=1}^{R} σ_{k} (u_{k}v_{k}^{T} ) = ∑_{k=1}^{R} σ_{k} u_{k}[v_{1k}, ⋯, v_{Nk}]

= [∑_{k=1}^{R}(σ_{k}v_{1k}) u_{k}, ⋯, ∑_{k=1}^{R}(σ_{k}v_{Nk}) u_{k}]  (A.361)

Each column vector of A can be expressed as a linear combination of the first R columns of U corresponding to the non-zero singular values:

c_{j} = ∑_{k=1}^{R}(σ_{k}v_{jk}) u_{k} (j = 1, ⋯, N),  (A.362)

Taking transpose on both sides of the SVD we also get:

(A^{T} ) = [r_{1}, ⋯, r_{M}] = VΣ^{T} U^{T}
= ∑_{k=1}^{R} σ_{k} (v_{k}u_{k}^{T} ) = ∑_{k=1}^{R} σ_{k} v_{k}[u_{1k}, ⋯, u_{Mk}]

= [∑_{k=1}^{R}(σ_{k}u_{1k}) v_{k}, ⋯, ∑_{k=1}^{R}(σ_{k}u_{Mk}) v_{k}]  (A.363)

i.e., each row vector of A can be expressed as a linear combination of the first R columns of V corresponding to the non-zero singular values:

r_{i} = ∑_{k=1}^{R}(σ_{k}u_{ik}) v_{k} (i = 1, ⋯, M)  (A.364)

We therefore see that the R columns of U and V corresponding to the non-zero singular values span respectively the column space C(A) and row space R(A):

C(A) = span(u_{1}, ⋯, u_{R}),  R(A) = span(v_{1}, ⋯, v_{R})  (A.365)

and the remaining M − R columns of U and N − R columns of V corresponding to the zero singular values span respectively N(A^{T}) orthogonal to C(A), and N(A) orthogonal to R(A):

N(A^{T}) = span(u_{R+1}, ⋯, u_{M}), N(A) = span(v_{R+1}, ⋯, v_{N}) (A.366)

Subspace  Definition  Dimension  Basis
C(A) ⊆ R^{M}  column space (image) of A  R  Rcolumns ofUcorresponding to non-zero singular values
N(A^{T}) ⊆ R^{M}  left null space of A^{T}  M − R  M − Rcolumns ofUcorresponding to zero singular values
R(A) ⊆ ℝ^{N}  row space (image) of A^{T}  R  Rcolumns ofVcorresponding to non-zero singular values
N(A) ⊆ ℝ^{N}  null space of A  N − R N − Rcolumns of V corresponding to zero singular values
R^{N} = N(A) ⊕ R(A)  domain of A  N  all N columns of V
R^{M} = N(A^{T}) ⊕ C(A)  codomain of A  M

all M columns of U

(A.367)

The SVD method can be used to find the pseudo-inverse of an M × N matrix A of rank R ≤ min(M, N):

A^{−} = VΣ^{−}U^{T}  (A.368)

where both A^{−} and

Σ^{−} = [ \begin{matrix}1/σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1/σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ]_{N×M}  (A.369)

are N × M matrices.

We further note that matrices U and V are related to each other by:

\{ \begin{matrix}A = UΣV^{T} \\ A^{−} = VΣ^{−}U^{T} \end{matrix}  or  \{ \begin{matrix}AV = UΣ \\ A^{−}U = VΣ^{−} \end{matrix}  (A.370)

or in vector form:

\{ \begin{matrix}Av_{i} = σ_{i}u_{i} (i = 1, ⋯, R) \\ A^{−}u_{i} = v_{i}/σ_{i} (i = 1, ⋯, R) \end{matrix}  \{ \begin{matrix}Av_{i} = 0 (i = R + 1, ⋯, N) \\ A^{−}u_{i} = 0 (i = R + 1, ⋯, M) \end{matrix}

(A.371)

indicating how the individual columns are related. We see that the last N − R columns v_{R+1}, ⋯, v_{N} of V form an orthogonal basis of N(A); and the last M − R columns u_{R+1}, ⋯, u_{M} of U form an orthogonal basis of N(A^{T}).

We now show that the optimal solution of the linear system Ax = b can be obtained based on the pseudo-inverse of A:

x_{svd} = A^{−}b = VΣ^{−}U^{T} b  (A.372)

Specially, if M = N = R, then A^{−} = A^{−1}, and x_{svd} = A^{−1}b is the unique and exact solution. In general when M ≠ N or R < min(M, N), the solution may not exist, or it may not be unique, but x_{svd} is still an optimal solution in two ways, from the perspective of both the domain and codomain of the linear mapping Ax, as shown below.

• In domain ℝ^{N}:

Pre-multiplying V^{T} = V^{−1} on both sides of the pseudo-inverse solution x_{svd} = VΣ^{−}U^{T} b given above, we get:

V^{T} x_{svd} = [v_{1}, ⋯, v_{N}]^{T} x_{svd} = Σ^{−}U^{T} b = Σ^{−}[u_{1}, ⋯, u_{M}]^{T} b, (A.373)

or in component form:

[ v_{1}^{T} x_{svd} ⋮ v_{R}^{T}x_{svd} v_{R+1}^{T}x_{svd} ⋮ v_{N}^{T}x_{svd} ] = [ \begin{matrix}1/σ_{1} & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1/σ_{R} \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ] [ u_{1}^{T} b ⋮ u_{R}^{T}b u_{R+1}^{T}b ⋮ u_{M}^{T}b ] = [ u_{1}^{T} b/σ_{1} ; ⋮ ; u_{R}^{T}b/σ_{R} ; 0 ; ⋮ ; 0 ]

(A.374)

The first R components v_{i}^{T} x_{svd} = u_{i}^{T} b/σ_{i} (i = 1, ⋯, R) are the projection of x_{svd} onto R(A) spanned by {v_{1}, ⋯, v_{R}} corresponding to the non-zero singular values, while the last N − R components v_{i}^{T} x_{svd} = 0 (i = R + 1, ⋯, N) are the projection of x_{svd} onto N(A) spanned by {v_{R+1}, ⋯, v_{N}} corresponding to the zero singular values. We see that the pseudo-inverse solution x_{svd} is entirely in R(A), containing no homogeneous component in N(A). In other words, x_{svd} has the minimum norm (closest to the origin) compared to any other possible solution x_{p}, such as those found previously based on the rref of A, containing a non-zero homogeneous component x_{h} ∈ N(A):

||x_{svd}|| ≤ ||x_{p}|| = ||x_{svd} + x_{h}||  (A.375)

As N(A) ⊥ R(A), x_{svd} ∈ R(A) is the projection of any such solution x_{p} onto R(A). The complete solution can be found as

x_{c} = x_{svd} + N(A) = x_{svd} + ∑_{j=R+1}^{N} c_{j}v_{j}  (A.376)

for any set of coefficients c_{j}.

• In codomain ℝ^{M}:

We now consider the result b_{svd} = Ax_{svd} produced by the pseudo-inverse solution x_{svd}, which, as a linear combination of the columns of A, is in its column space C(A):

b_{svd} = Ax_{svd} = AA^{−}b = UΣV^{T} VΣ^{−}U^{T} b = UΣΣ^{−}U^{T} b (A.377)

Pre-multiplying U^{T} = U^{−1} on both sides we get:

U^{T} b_{svd} = ΣΣ^{−}U^{T} b  (A.378)

or in component form:

[ u_{1}^{T} b_{svd} ⋮ u_{R}^{T}b_{svd} u_{R+1}^{T}b_{svd} ⋮ u_{M}^{T}b_{svd} ] = [ \begin{matrix}1 & & & & & & & & & 0 \\ & . & . & . \\ & & & & 1 \\ & & & & & 0 \\ & & & & & & . & . & . \\ 0 & & & & & & & & & 0 \end{matrix} ] [ u_{1}^{T} b ⋮ u_{R}^{T}b u_{R+1}^{T}b ⋮ u_{M}^{T}b ] = [ u_{1}^{T} b ; ⋮ ; u_{R}^{T}b ; 0 ; ⋮ ; 0 ]

(A.379)

The first R components u_{i}^{T} b_{svd} = u_{i}^{T} b (i = 1, ⋯, R) are the projection of b_{svd} onto C(A) spanned by {u_{1}, ⋯, u_{R}} corresponding to the non-zero singular values, while the last M − R components u_{i}^{T} b_{svd} = 0 (i = R + 1, ⋯, M) are the projection of b_{svd} onto N(A^{T}) spanned by {u_{R+1}, ⋯, u_{M}} corresponding to the zero singular values. We see that b_{svd} and b have the same projection onto C(A). If b ∈ C(A), i.e., u_{i}^{T} b = 0 for all i = R + 1, ⋯, M, then b_{svd} = Ax_{svd} = b and x_{svd} is an exact solution. But if b ∉ C(A), as it contains non-zero components in N(A^{T}), then no solution exists. However, as N(A^{T}) ⊥ C(A), b_{svd} ∈ C(A) is the projection of b onto C(A), the error ||b − b_{svd}|| is minimized when compared with that of any other approximate solution x_{p} ≠ x_{svd}:

||b − b_{svd}|| = ||b − Ax_{svd}|| ≤ ||b − Ax_{p}||  (A.380)

In this sense, x_{svd} is the optimal.

Summarizing the two aspects above, we see that the pseudo-inverse solution x_{svd} is optimal in the sense that both its norm ||x_{svd}|| and its error ||Ax_{svd} −b|| are minimized.

• If the solution x_{svd} is not unique because N(A) ≠ ∅, then the complete solution can be found by adding the entire null space to it: x_{c} = x_{svd} + N(A).

• If no solution exists because b ∉ C(A), then x_{svd} is the optimal approximate solution with minimum error.

Example A.8 Given the same system considered in previous examples

Ax = [ \begin{matrix}1 & 2 & 3 & 4 \\ 4 & 3 & 2 & 1 \\ −2 & 1 & 4 & 7 \end{matrix} ] [ \begin{matrix}x_{1} \\ x_{2} \\ x_{3} \\ x_{4} \end{matrix} ] = [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ]  (A.381)

we will now solve it using the pseudo-inverse method. We first find SVD of A in terms of the following matrices:

U = [u_{1}, u_{2}, u_{3}] = [ \begin{matrix}−0.535 & −0.218 & 0.817 \\ −0.267 & −0.873 & −0.408 \\ −0.802 & 0.436 & −0.408 \end{matrix} ]  (A.382)

V = [v_{1}, v_{2}, v_{3}, v_{4}] = [ \begin{matrix}0.000 & −0.837 & 0.374 & −0.400 \\ −0.267 & −0.478 & −0.797 & 0.255 \\ −0.535 & −0.120 & 0.472 & 0.691 \\ −0.802 & 0.239 & −0.049 & −0.546 \end{matrix} ] (A.383)

Σ = [ \begin{matrix}10.0 & 0 & 0 & 0 \\ 0 & 5.477 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{matrix} ] ,  Σ^{−} = [ \begin{matrix}0.1 & 0 & 0 \\ 0 & 0.183 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix} ]  (A.384)

A^{−} = VΣ^{−}U^{T} = [ \begin{matrix}0.000 & −0.837 & 0.374 & −0.400 \\ −0.267 & −0.478 & −0.797 & 0.255 \\ −0.535 & −0.120 & 0.472 & 0.691 \\ −0.802 & 0.239 & −0.049 & −0.546 \end{matrix} ] [ \begin{matrix}0.100 & 0 & 0 \\ 0 & 0.183 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix} ] [ \begin{matrix}−0.535 & −0.267 & 0.802 \\ −0.218 & −0.873 & 0.436 \\ 0.817 & −0.408 & −0.408 \end{matrix} ]

= [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ]

The particular solution of the system Ax = b with b = [3, 2, 4]^{T} ∈ C(A) is

x_{svd} = A^{−}b = [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ] [ \begin{matrix}3 \\ 2 \\ 4 \end{matrix} ] = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]

= −0.535 [ \begin{matrix}0.000 \\ −0.267 \\ −0.535 \\ −0.802 \end{matrix} ] − 0.120 [ \begin{matrix}−0.837 \\ −0.478 \\ −0.120 \\ 0.239 \end{matrix} ] = c_{1}v_{1} + c_{2}v_{2}

which is in R(A) spanned by the first R = 2 columns v_{1} and v_{2}, perpendicular to N(A) spanned by the last N − R = 2 columns v_{3} and v_{4}. Note that this solution x_{svd} = [0.1, 0.2, 0.3, 0.4]^{T} is actually the first component x′_{p} ∈ R(A) of the particular solution x_{p} = [−1, 2, 0, 0] found in the previous section by Gauss-Jordan elimination, which is not in R(A). Adding the null space to the particular solution, we get the complete solution:

x_{c} = x_{svd}+N(A) = x_{p}+c_{1}v_{3}+c_{2}v_{4} = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]+c_{1} [ \begin{matrix}0.374 \\ −0.797 \\ 0.472 \\ −0.049 \end{matrix} ]+c_{2} [ \begin{matrix}−0.400 \\ 0.255 \\ 0.691 \\ −0.546 \end{matrix} ]

(A.385)

If the right-hand side is replaced by b = [1, 3, 5]^{T} ∉ C(A), no solution exists. However, we can still find the pseudo-inverse solution as the optimal approximate solution:

x_{svd} = A^{−}b = [ \begin{matrix}0.033 & 0.133 & −0.067 \\ 0.033 & 0.083 & −0.017 \\ 0.033 & 0.033 & 0.033 \\ 0.033 & −0.017 & 0.083 \end{matrix} ] [ \begin{matrix}1 \\ 3 \\ 5 \end{matrix} ] = [ \begin{matrix}0.1 \\ 0.2 \\ 0.3 \\ 0.4 \end{matrix} ]  (A.386)

which is the same as the solution for b = [3, 2, 4]^{T}, indicating [3, 2, 4]^{T} happens to be the projection of [1, 3, 5]^{T} onto C(A) spanned by u_{1} and u_{2}. The result produced by x_{svd} is b′ = Ax_{svd} ∈ C(A) is the projection of b onto C(A), with a minimum error distance ||e|| = ||b − b′||, indicating x_{svd} is the optimal approximate solution.

Homework 3:

1. Use Matlab function svd(A) to carry out the SVD of the coefficient matrix A of the linear system in Homework 2 problem 3,

A = [ \begin{matrix}−1 & 3 & 4 & 1 \\ 2 & −4 & 3 & 2 \\ 1 & −1 & 7 & 3 \end{matrix} ]  (A.387)

Find U, V, Σ. Verify that U and V are orthogonal, i.e., U^{T} U = I and V^{T} V = I.

2. Obtain Σ^{−}, i.e, find the transpose of Σ and then replace each singular value by its reciprocal, verify that ΣΣ^{−} = I and Σ^{−}Σ = I. Then find the pseudoinverse A^{−} = UΣ^{−}V^{T}.

3. Use Matlab function pinv(A) to find the pseudo-inverse Σ^{−} and A^{−}. Compare them to what you obtained in the previous part.

4. Identify the bases of the four subspaces R(A), N(A), C(A) = R(A^{T}), and N(A^{T}) based on U and V. Verify that these bases and those you found previously (problem 3 of Homework 2) span the same spaces, i.e., the basis vectors of one basis can be written as a linear combination of those of the other, for each of the four subspaces.

5. Use the pseudo-inverse A^{−} found above to solve the system Ax = b. Find the two particular solutions x_{p1} and x_{p2} corresponding to two different right-hand side b_{1} = [1, 2, 3]^{T} and b_{2} = [2, 3, 2]^{T}.

6. How are the two results x_{p1} and x_{p2} related to each other? Give your explanation. Why can’t you find a solution when b = [2, 3, 2]^{T} but SVD method can?

7. Verify that x_{p} ⊥ N(A). Also find the error (or residual) e = b−Ax_{p} for each of the two results above. Verify that e ⊥ C(A), if e ≠ 0.

8. In Homework 2 you used row reduction method to solve the system Ax = b_{1} and you should have found a particular solution x_{1} = [5, 2, 0, 0, ]^{T}. Also it is obvious to see that another solution is x_{2} = [0, 0, 0, 1]^{T}. Show that the projection of these solutions onto R(A) spanned by the first two columns of V is the same as the particular solution x_{p} found by the SVD method.

9. Show that b_{1} = [1, 2, 3]^{T} is the projection of b_{2} = [2, 3, 2]^{T} onto C(A), the 2-D subspace spanned by the first two columns of U. Can you also show that it is the projection of b onto C(A) spanned by the basis you obtained in Homework 2 (not necessarily orthogonal)?

10. Give an expression of the null space N(A), and then write the complete solution in form of x_{c} = x_{p}+N(A). Verify any complete solution so generated satisfies Ax_{c} = b_{1}.

A.7 Taylor Series Expansion

• Single-variable functions:

A single-variable function f(x) can be expanded around a given point x by the Taylor series:

f(x + δx) = f(x) + f′(x)δx + \frac{1}{2!}f′′(x)δx^{2} + ⋯ + \frac{1}{n!}f^{(n)}(x)δx^{n} + O(δx^{n+1})

(A.388)

where O(δx^{n+1}) denotes all the remaining terms with order higher than n in the infinite series. If δx is much smaller than 1, O(δx^{n+1}) can be neglected so that the function f(x) can be approximated by the first n terms. For example, f(x) can be approximated as a quadratic function with a third order error term O(δx^{3}) when n = 2, or a linear function with a second order error term O(δx^{2}) when n = 1:

f(x + δx) = f(x) + f′(x)δx + \frac{1}{2!}f′′(x)δx^{2} + O(δx^{3})
≈ f(x) + f′(x)δx + \frac{1}{2!}f′′(x)δx^{2}
≈ f(x) + f′(x)δx  (A.389)

• Multi-variable scalar-valued functions:

A multi-variable function f(x_{1}, ⋯, x_{n}) = f(x) can also be expanded by the Taylor series:

f(x + δx) = f(x) + ∑_{j=1}^{n} \frac{∂f(x)}{∂x_{j}}δx_{j} + \frac{1}{2} ∑_{i=1}^{n} ∑_{j=1}^{n} \frac{∂^{2}f(x)}{∂x_{i}∂x_{j}} δx_{i}δx_{j}

+ \frac{1}{3!} ∑_{i=1}^{n} ∑_{j=1}^{n} ∑_{k=1}^{n} \frac{∂^{3}f(x)}{∂x_{i}∂x_{j}∂x_{k}} δx_{i}δx_{j}δx_{k} + ⋯  (A.390)

which can be expressed in vector form as:

f(x + δx) = f(x) + g_{f}^{T} δx + \frac{1}{2}δx^{T} H_{f} δx + ⋯  (A.391)

where δx = [δx_{1}, ⋯, δx_{n}]^{T}, while g_{f} and H_{f} are the gradient vector and the Hessian matrix for the first and second order derivatives of the function, respectively:

g_{f}(x) = ▽f(x) = \frac{d}{dx}f(x) = [ ⋮_{\frac{∂fx)}{∂x_{n}}}^{\frac{∂fx)}{∂x_{1}}} ]_{n×1} ,  (A.392)

H_{f}(x) = \frac{d}{dx}g_{f}(x) = \frac{d^{2}}{dx^{2}} f(x) = [ \begin{matrix}\frac{∂^{2}f(x)}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x)}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{n}^{2}} \end{matrix} ]_{n×n} = H_{f}^{T}

(A.393)

Note that H_{f}^{T} is symmetric as ∂^{2}f(x)/∂x_{i}∂x_{j} = ∂^{2}f(x)/∂x_{j}∂x_{i}.

When δx is small (i.e., ||δx|| is small), then f(x+δx) can be approximated by a quadratic function with a third order error term

f(x + δx) = f(x) + g_{f}^{T} δx + \frac{1}{2}δx^{T} H_{f} δx + O(||δx||^{3})  (A.394)

or even a linear function with a second order error term

f(x + δx) = f(x) + g_{f}^{T} δx + O(||δx||^{2})  (A.395)

• Multi-variable vector-valued functions:

A set of m multi-variable functions f_{1}(x), ⋯, f_{m}(x) can be expressed as a vector function

f(x) = [f_{1}(x), ⋯, f_{m}(x)]^{T}  (A.396)

The Taylor expansion of the ith component is:

f_{i}(x + δx) = f_{i}(x) + g_{i}^{T} δx + \frac{1}{2}δx^{T} H_{i} δx + O(||δx||^{3}) (i = 1, ⋯, m)

(A.397)

The first two terms of these m components can be written in vector form:

f(x+δx) ≈ [ f_{1}(x) ; ⋮ ; f_{m}(x) ]+[ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} & ⋯ & \frac{∂f_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{m}}{∂x_{1}} & ⋯ & \frac{∂f_{m}}{∂x_{n}} \end{matrix} ] [ δx_{1} ; ⋮ ; δx_{n} ] = f(x)+J_{f} (x) δx

(A.398)

where J_{f} (x) is the Jacobian matrix defined over the vector function f(x):

J_{f} (x) = \frac{d}{dx}f(x) = [\frac{∂f}{∂x_{1}} , ⋯, \frac{∂f}{∂x_{n}} ] = [ \begin{matrix}\frac{∂f_{1}}{∂x_{1}} & ⋯ & \frac{∂f_{1}}{∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂f_{m}}{∂x_{1}} & ⋯ & \frac{∂f_{m}}{∂x_{n}} \end{matrix} ]_{m×n} (A.399)

Note that the Jacobian matrix is not necessarily square as m and n may not be the same. The 2nd and higher order terms can no longer expressed in matrix.

Note that the Hessian matrix of a function f(x) can be obtained as the Jacobian matrix of the gradient vector of f(x):

g_{f}(x) = [ ⋮_{\frac{∂f}{∂x_{n}}}^{\frac{∂f}{∂x_{1}}} ]

J_{g}(x) = \frac{d}{dx}g_{f}(x) = H_{f} (x) = [ \begin{matrix}\frac{∂^{2}f}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f}{∂x_{n}^{2}} \end{matrix} ] (A.400)

A.8 Miscellaneous

A.8.1 Centering Matrix

The centering matrix is defined as

C_{n} = I_{n} − \frac{1}{n}11^{T}  (A.401)

where 1 = [1, ⋯, 1]^{T} and 11^{T} is an n × n matrix with all components equal to 1. when n = 1, 2 3, we have

C_{1} = 0;  C_{2} = [ \begin{matrix}1 & 0 \\ 0 & 1 \end{matrix} ] − \frac{1}{2} [ \begin{matrix}1 & 1 \\ 1 & 1 \end{matrix} ] = \frac{1}{2} [ \begin{matrix}1 & −1 \\ −1 & 1 \end{matrix} ];  (A.402)

C_{3} = [ \begin{matrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ] − \frac{1}{3} [ \begin{matrix}1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{matrix} ] = \frac{1}{3} [ \begin{matrix}2 & −1 & −1 \\ −1 & 2 & −1 \\ −1 & −1 & 2 \end{matrix} ]  (A.403)

Premultiplying C_{n} to an n-D vector v = [v_{1}, ⋯, v_{n}]^{T}, we get

C_{n}v = v − \frac{1}{n} (11^{T} ) v = v − \frac{1}{n}11^{T} v = v − \frac{1}{n}1 ∑_{i=1}^{n} v_{i} = v − v̄1 (A.404)

where v̄ is the mean of all components of v:

v̄ = \frac{1}{n}1^{T} v = \frac{1}{n} ∑_{i=1}^{n} v_{i}  (A.405)

We see that the mean of v is removed from the resulting vector C_{n}v.

Example A.9 v = [3, −2, 5]^{T}, with mean v̄ = (3 − 2 + 5)/3 = 2,

u = C_{3}v = \frac{1}{3} [ \begin{matrix}2 & −1 & −1 \\ −1 & 2 & −1 \\ −1 & −1 & 2 \end{matrix} ] [ \begin{matrix}3 \\ −2 \\ 5 \end{matrix} ] = [ \begin{matrix}3 \\ −2 \\ 5 \end{matrix} ] − [ \begin{matrix}2 \\ 2 \\ 2 \end{matrix} ] = [ \begin{matrix}1 \\ −4 \\ 3 \end{matrix} ]

The mean of the resulting vector u is ū = 0. In particular, consider applying C_{n} to the following two particular vectors:

• v = 1

C_{n}v = C_{n}1 = [I − \frac{1}{n}11^{T} ] 1 = 1 − \frac{1}{n}11^{T} 1 = 1 − \frac{1}{n}1n = 0 (A.407)

Treating C_{n}1 = 0 1 as an eigenequation of C_{n}, we see that λ = 0 and v = 1 are respectively the eigenvalue and corresponding eigenvector of C_{n}.

• v = [v_{1}, ⋯, v_{n}] with a zero mean ∑_{i=1}^{n} v_{i} = 1^{T} v = 0 (with n − 1 degrees of freedom)

C_{n}v = [I − \frac{1}{n}11^{T} ] v = v − \frac{1}{n}11^{T} v = v  (A.408)

As C_{n}v = 1 v, we see that λ = 1 and any v satisfying 1^{T} v = 0 are respectively the eigenvalue of multiplicity n − 1 and the corresponding eigenvector of C_{n}.

The centering matrix C_{n} has the following properties:

• Idempotence: C_{n}^{k} = C_{n}

C_{n}^{2} = C_{n}C_{n} = (I_{n} − \frac{1}{n}11^{T} ) (I_{n} − \frac{1}{n}11^{T} ) = I_{n} − \frac{2}{n}11^{T} + \frac{n}{n^{2}} 11^{T} = C_{n}

(A.409)

Once the mean of vector v is removed, its mean is zero and subsequent removal of mean has not effect.

• C_{n} has eigenvalue λ = 1 of multiplicity n − 1 and eigenvalue λ = 0 of multiplicity 1.

• C_{n} is singular, its inverse does not exist. Once the mean of vector v is removed, it cannot be reconstructed by an inverse process.

A.8.2 Normal Direction of a Plane

Consider the following in a d-dimensional space.

• A point is represented as a column vector x = [x_{1}, ⋯, x_{d}]^{T} containing its d coordinates;

• A surface (or hypersurface if d > 3) is represented by an equation f(x_{1}, ⋯, x_{d}) = f(x) = 0. Alternatively, by solving the equation above for one of the variables, we get x_{d} = g(x_{1}, ⋯, x_{d−1}), representing the height along the direction of x_{d} of a surface defined over a d − 1 dimensional space.

• A plane (or hyperplane if d > 3) is a special surface described by a linear equation:

f(x) = w^{T} x + b = ∑_{i=1}^{d} w_{i}x_{i} + b = 0  (A.410)

Without loss of generality, we can assume w is normalized with length ||w|| = \sqrt{w^{T} w} = 1 (as otherwise we can divide the equation above by ||w||).

• The d-dimensional space is divided by the plane into two regions:

R_{+} = {x | w^{T} x + b > 0}, and R_{−} = {x | w^{T} x + b < 0} (A.411)

• The normal direction of a plane f(x) = w^{T} x + b = 0 is w, i.e., the projection of any point x on the plane onto normal direction w is constant:

p_{w}(x) = w^{T} x = −b  (A.412)

• Solving the linear equation w^{x} + b = 0 for any of the d variables, e.g., x_{d}, we get an alternative representation for the plane, as a function defined over the d-1 dimensional space spanned by the remaining d − 1 variables {x_{1}, ⋯, x_{d−1}}:

x_{d} = g(x_{1}, ⋯, x_{d−1}) = −\frac{1}{w_{d}} (∑_{i=1}^{d−1} w_{i}x_{i} + b)  (A.413)

In particular, when x_{1} = ⋯ = x_{d−1} = 0, we get x_{d} = −b/w_{d}, the intercept of the plane on the axis of x_{d}.

• When the plane is represented in the form of x_{d} = g(x_{1}, ⋯, x_{d−1}) = −(∑_{i=1}^{d−1} w_{i}x_{i}+ b)/w_{d}, the normal direction is represented the following, a scaled version of w:

[−\frac{w_{1}}{w_{d}} , ⋯, −\frac{w_{d−1}}{w_{d}} , −1]^{T} = −\frac{1}{w_{d}} [w_{1}, ⋯, w_{d}]^{T} = −\frac{1}{w_{d}} w  (A.414)

For example, the following equation defines a plane in 3-D space spanned by x, y and z:

f(x, y, z) = ax + by + cz + d = x + 2y + 3z + 4 = 0

where a = 1, b = 2, c = 3 and d = 4. Solving for z we get

z = g(x, y) = −\frac{1}{c}(ax + by + d) = −\frac{1}{3}(x + 2y + 4)

When x = y = 0, we get z = −c/d = −4/3, the intercept of the plane on z-axis. The normal direction of the plane is composed of the coefficients of the three variable x, y and z in f(x, y, z):

n = [a, b, c]^{T} = [1, 2, 3]^{T}  (A.415)

or the coefficients of the two variables x and y of g(x, y), with the last component −1:

n = [−\frac{a}{c}, −\frac{b}{c}, −1]^{T} = −\frac{1}{c}[a, b, c]^{T} = −\frac{1}{3}[1, 2, 3]  (A.416)

A.8.3 Convexity

We first consider some concepts regarding convexity:

• Convex set or convex region: a region in an N-D space in which every point on the straight line between any pair of two points in the region is also inside the region.

• Convex hull or convex envelope, convex closure of a shape: the smallest convex set containing the shape.

• Convex function: a real-valued function f(x) with the property that its epigraph or supergraph, the region above its graph, is a convex set.

• Concave function: if f(x) is convex, then −f(x) is concave.

A multivariate function f(x) is convex if and only if

f(λx_{1} + (1 − λ)x_{2}) ≤ λf(x_{1}) + (1 − λ)f(x_{2})  (A.417)

Equivalently, f(x) is concave if and only if

f(λx_{1} + (1 − λ)x_{2}) ≥ λf(x_{1}) + (1 − λ)f(x_{2})  (A.418)

The convex function is strict if equality in the relationship never holds. Geometrically, the line segment connecting two points (x_{1}, f(x_{1})) and (x_{2}, f(x_{2})) is above the graph of the convex function f(x).

Convexity plays an important role in optimization problems, due to its nice property that if a function f(x) is differentiable and strictly convex, then any point x^{∗} with a zero gradient g_{f}(x^{∗}) = 0 is the unique global minimum of the function. Minimizing a convex objective function, or equivalently maximizing a concave objective function, over a convex domain is called convex optimization.

Here are some examples and properties of convex functions:

• Single-variable functions: x^{2}, |x|, e^{ax}, − log(x)

• Any norm of a function or vector is convex. The intersection of the epigraphs is convex.

• The intersection of two convex sets is convex. Also,

• Affine function: f(x) = w^{T} x + b is both convex and concave but not strict, as both Eq. (A.417) and Eq. (A.418) become equality when applied to the function.

• If function f(x) is convex, then a f(x) + b is also convex, if a ≥ 0 so that the convexity is preserved.

• If two functions f(x) and g(x) are both convex, then their linear combination is also convex if the coefficients are no negative:

h(x) = af(x) + bg(x), a, b ≥ 0  (A.419)

This can be proven by applying the definition directly to h(x).

• A linear combination of n convex function is also convex if all coefficients are nonnegative:

f(x) = ∑_{i=1}^{n} w_{i}f_{i}(x), w_{1}, ⋯, w_{n} ≥ 0  (A.420)

• Quadratic functions: f(x) = x^{T} Qx + w^{T} x + b is – convex if Q ≥ 0 is positive semidefinite;

– strictly convex if Q > 0 is positive definite;

– concave if Q ≤ 0 is negative semidefinite;

– strictly concave if Q < 0 is negative definite;

If the Hessian matrix H_{f}(x), the second order derivative, of f(x) exists and is positive semidefinite, i.e., H_{f}(x) ≥ 0 for its entire domain, then f(x) is convex. As simple examples in 1-D space, f_{1}(x) = x^{2} and f_{2}(x) = e^{x} are both convex, and their Hessian, the second order derivatives, f′′_{1}(x) = 2 and f′′_{2}(x) = e^{x} are both greater than zero.

Specifically, let x_{1} and x_{2} be any two points inside the domain of f(x). Then f(x) is convex if for any c ∈ [0, 1] the following holds:

f(cx_{1} + (1 − c)x_{2}) ≤ c f(x_{1}) + (1 − c)f(x_{2})  (A.421)

where cx_{1} + (1 − c)x_{2} is any point along the line segment between x_{1} and x_{2}, at which the function value is less than the linear interpolation of the two function values f(x_{1}) and f(x_{2}). If the equality does not hold, then f(x) is strictly convex. We can also define a concave function as the negative of a convex function. If f(x) is convex, then g(x) = −f(x) is concave, for which the following inequality holds:

g(cx_{1} + (1 − c)x_{2}) ≥ c g(x_{1}) + (1 − c)g(x_{2})  (A.422)

A convex function f(x) has the property that any local minimum is also a global minimum. Moreover, if f(x) is strictly convex, then it has a single global minimum.

A.8.4 Jensen Inequality

For a convex function g(x), the following holds:

g (∑_{i=1}^{n} λ_{i}x_{i}) ≤ ∑_{i=1}^{n} λ_{i}f(x_{i})  (A.423)

For a concave function g(x), the following holds:

g (∑_{i=1}^{n} λ_{i}x_{i}) ≥ ∑_{i=1}^{n} λ_{i}g(x_{i})  (A.424)

Proof:

Let λ_{1} and λ_{2} be two arbitrary nonnegative real numbers satisfying λ_{1}+λ_{2} = 1. Then by the definition of convexity, we have

f(λ_{1}x_{1} + λ_{2}x_{2}) ≤ λ_{1}f(x_{1}) + λ_{2}f(x_{2})  (A.425)

This results of two points can be generalized to n points by induction. We assume the inequality holds for k points:

f (∑_{i=1}^{k} λ_{i}x_{i}) ≤ ∑_{i=1}^{k} λ_{i}f(x_{i})  (A.426)

Now consider the case of k+1 terms with coefficients satisfying λ_{1}+⋯+λ_{k+1} = 1:

f (∑_{i=1}^{k+1} λ_{i}x_{i}) = f (λ_{1}x_{1} + (1 − λ_{1}) ∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i})

≤ λ_{1}f(x_{1}) + (1 − λ_{1})f (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i})  (A.427)

But as

∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} = 1  (A.428)

we have the following due to the assumption above:

f (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} x_{i}) ≤ ∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} f(x_{i})  (A.429)

Substituting this back we get the desired inequality:

f (∑_{i=1}^{k+1} λ_{i}x_{i}) ≤ λ_{1}f(x_{1}) + (1 − λ_{1}) (∑_{i=2}^{k+1} \frac{λ_{i}}{1 − λ_{1}} f(x_{i})) = ∑_{i=1}^{k+1} λ_{i}f(x_{i}) (A.430)

If x is a random vector that takes any of the n values x_{i} with probability P_{i} satisfying P_{1} + ⋯ + P_{n} = 1, then we have

f(E(x)) = f (∑_{i=1}^{k} x_{i}P_{i}) ≤ ∑_{i=1}^{k} f(x_{i})P_{i} = E(f(x))  (A.431)

This result can be generalized to the case where x is a random vector taking continuous values with E[x] = ∫ x p(x)dx.

A.8.5 Sensitivity and Conditioning

An important concern in numerical analysis is how sensitive the result produced by an algorithm is with respect to the noise in the input data (e.g., observational error) and computational errors (e.g., truncation error). To address this issue, we model a generic algorithm treated as a function with both input and output most generally defined (e.g., real or complex, scalar, vector, matrix, function, etc.), and describe its sensitivity to noise as below:

• A function is well conditioned (well behaved) if a small change in the input causes a small change in the output.

• A function is ill conditioned (ill behaved) if a small change in the input causes a large change in the output.

How well or ill a function is conditioned can be measured quantitatively by the absolute or relative condition numbers defined below:

• The absolute condition number is the upper bound of the ratio between the change in output and change in input:

κ̂ ≥ \frac{||change in output||}{||change in input||}  (A.432)

• The relative condition number is the upper bound of the ratio between the relative change in output and relative change in input:

κ ≥ \frac{||change in output||/||output||}{||change in input||/||input||}  (A.433)

where ||x|| denotes the norm of x. As the relative condition number compares the normalized changes in both input and output, its value is invariant to the specific units used to measure them. Obviously the smaller the condition number, the better the function is conditioned.

Consider specifically the condition numbers of some different systems.

System with Scalar input and output:

We first consider evaluating a scalar function y = f(x) given x. The condition number is:

κ̂ = lim_{ϵ→0} sup_{|δx|≤ϵ} \frac{|δf(x)|}{|δx|},  κ = lim_{ϵ→0} sup_{|δx|≤ϵ} \frac{|δf(x)|/|f(x)|}{|δx|/|x|},  (A.434)

Here sup is the symbol for supremum (least upper bound) and |x| is either the absolute value of x if it is real or the modulus of x if it is complex. Note that in general the condition number is a function of x.

When δx is small, f(x + δx) can be approximated by the first two terms of its Taylor series:

f(x + δx) = f(x) + f′(x)δx + f′′(x)\frac{δx^{2}}{2} + ⋯ ≈ f(x) + f′(x)δx (A.435)

then

δy = f(x + δx) − f(x) ≈ f′(x)δx  (A.436)

Taking the absolute value or modulus on both sides, we get

|δy| ≈ |f′(x)δx| ≤ |f′(x)| |δx|  (A.437)

and the absolute condition number is:

κ̂ ≈ |δy|/|δx| ≈ |f′(x)|  (A.438)

Dividing both sides by |y| = |f(x)|, we further get

\frac{|δy|}{|y|} ≈ \frac{|f′(x)| |δx|}{|f(x)|} = \frac{|x| |f′(x)|}{|f(x)|} \frac{|δx|}{|x|}  (A.439)

and the relative condition number:

κ = \frac{|δy|/|y|}{|δx|/|x|} ≈ \frac{|x| |f′(x)|}{|f(x)|} = \frac{|f′(x)|}{|f(x)|/|x|}  (A.440)

We next consider solving the equation f(x) = 0 to find x treated as the output that satisfies y = f(x) = 0 treated as the input. The problem can be converted into the evaluation of the function x = g(y) = f^{−1}(y) at y = 0. The absolute condition number of this function g(y) = f^{−1}(y) is:

κ̂ = |g′(y)| = |\frac{d}{dy} [f^{−1}(y)]| = \frac{1}{|f′(x)|}  (A.441)

which is the reciprocal of the absolute condition number |f′(x)| of the previous evaluating problem, evaluated at the root x^{∗} satisfying f(x^{∗}) = 0. We see that the larger |f′(x)|, or the smaller g′(y) = 1/|f′(x)|, the better conditioned the problem is.

In the figure below shows two functions |f′_{1}(x)| > |f′_{2}(x)| for all x. For evaluating function y = f(x), f_{1}(x) is more ill-conditioned than f_{2}(x), but for solving f(x) = 0, f_{1}(x) is more well-conditioned than f_{2}(x).

Consider as an example the following two problems both associated with a linear function y = f(x) = ax + b:

• Evaluation of y given x:

The condition number is κ = δy/δx. If x is changed to x + δx due to noise, then y becomes ax + b + aδx = y + δy, where δy = aδx, i.e., κ = a. The larger a, the more ill-conditioned the system is.

• Solving y = f(x) = 0 for x given y = 0:

The condition number is κ = δx/δy. The given equation can be solved for x:

x = g(y) = \frac{1}{a}(y − b)  (A.442)

If y is changed to y+δy due to noise, then x becomes (y−b)/a+δy/a = x+ δx, where δx = δy/a, i.e., κ = 1/a. The smaller a, the more ill-conditioned the system is.

Systems with vector input and output

The discussions above can be generalized to a function y = f(x) with vector input x = [x_{1}, ⋯, x_{N}]^{T} and vector output y = [y_{1}, ⋯, y_{M}]^{T}. Again consider approximating the function by the first two terms in the Taylor series:

f(x + δx) ≈ f(x) + J_{f}(x)δx  (A.443)

where J_{f}(x) is the Jacobian matrix of the function. We get

||δy|| = ||f(x + δx) − f(x)|| ≈ ||J_{f}(x)δx|| ≤ ||J_{f}(x)|| ||δx||  (A.444)

and the absolute condition number:

κ̂ = \frac{||δy||}{||δx||} ≈ ||J_{f}(x)||  (A.445)

Dividing both sides of the equation above by ||y|| = ||f(x)|| we further get

\frac{||δy||}{||y||} ≈ \frac{||J_{f}(x)δx||}{||f(x)||} ≤ \frac{||x|| ||J_{f}(x)||}{||f(x)||} \frac{||δx||}{||x||}  (A.446)

and we can find the relative condition number as

κ = \frac{||δy||/||y||}{||δx||/||x||} ≈ \frac{||x|| ||J_{f}(x)||}{||f(x)||}  (A.447)

Consider as an example the following two problems both associated with a linear function:

y = f(x) = Ax  (A.448)

• Evaluating y = Ax given x:

The Taylor series of this linear function is

f(x + δx) = A(x + δx) = Ax + J_{f} δx = Ax + δAx  (A.449)

i.e., δy = Aδx and ||δy|| ≤ ||A|| ||δx||. We get the upper bound of the absolute condition number:

κ̂(A) = \frac{||δy||}{||δx||} ≤ ||A||  (A.450)

The relative condition number of this linear system is

κ(A) = \frac{||x|| ||J_{f}(x)||}{||f(x)||} = ||A||\frac{||x||}{||Ax||}  (A.451)

We also have

||x|| = ||A^{−1}Ax|| ≤ ||A^{−1}|| ||Ax||, i.e., ||Ax|| ≥ \frac{||x||}{||A^{−1}||} (A.452)

Substituting this into the expression of κ(A) above we get its upper bound:

κ(A) = ||A||\frac{||x||}{||Ax||} ≤ ||A|| ||A^{−1}||\frac{||x||}{||x||} = ||A|| ||A^{−1}||  (A.453)

Note that κ is no less than 1:

κ(A) = ||A^{−1}|| ||A|| ≥ ||A^{−1}A|| = ||I|| = 1  (A.454)

• Solving Ax = y for x as output given y as input:

We assume A is a full-rank square matrix and therefore invertible, so that there exists a unique solution: x = g(y) = A^{−1}y. The absolute condition number is κ̂(A) = ||J_{g}(x)|| = ||A^{−1}||.

To find the relative condition number κ(A), consider δx in the output x caused by δA as well as δy:

(A + δA)(x + δx) = Ax + Aδx + δAx + δAδx = y + δy  (A.455)

Subtracting Ax = y from both sides we get

Aδx + δAx + δAδx = δy  (A.456)

i.e.,

δx = A^{−1}(δy − δAx − δAδx)  (A.457)

Taking norm on both sides, we get

||δx|| ≤ ||A^{−1}|| ||δy − (δAx + δAδx)||
≤ ||A^{−1}|| (||δy|| + ||(δAx + δAδx)||)
≤ ||A^{−1}|| (||δy|| + ||δA|| ||x|| + ||δA|| ||δx||) (A.458)

where we have used ||a − b|| ≤ ||a|| + ||b||, ||a + b|| ≤ ||a|| + ||b||, and ||Ax|| ≤ ||A|| ||x||.

Subtracting ||A^{−1}|| ||δA|| ||δx|| from both sides

(1 − ||A^{−1}|| ||δA||) ||δx|| ≤ ||A^{−1}|| (||δy|| + ||δA|| ||x||)  (A.459)

and dividing both sides by (1 − ||A^{−1}|| ||δA||) ||y|| (assuming not zero), we get

\frac{||δx||}{||x||} ≤ \frac{||A^{−1}||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||x||} + ||δA||) = \frac{||A^{−1}|| ||A||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||x|| ||A||} + \frac{||δA||}{||A||})
≤ \frac{||A^{−1}|| ||A||}{1 − ||A^{−1}|| ||δA||} (\frac{||δy||}{||y||} + \frac{||δA||}{||A||}) = \frac{κ(A)}{1 − κ(A)||δA||/||A||} (\frac{||δy||}{||y||} + \frac{||δA||}{||A||}) (A.460)

where we have used ||y|| = ||Ay|| ≤ ||A|| ||x||, and κ(A) = ||A^{−1}|| ||A||.

In particular,

– if δA = 0,

\frac{||δx||}{||x||} ≤ κ(A)\frac{||δy||}{||y||}  (A.461)

– if δy = 0,

\frac{||δx||}{||x||} ≤ \frac{κ(A)}{1 − κ(A)||δA||/||A||} \frac{||δA||}{||A||}  (A.462)

We see that the relative condition number is κ(A) = ||A|| ||A^{−1}||, same for both problems of evaluation y = Ax and solving equation Ax = y.

We further consider the spectral norm defined as the greatest singular value of the matrix ||A|| = σ_{max}. Then ||A^{−1}|| = 1/σ_{min} is the reciprocal of the smallest singular value of A. Now the relative condition number is

κ(A) = ||A|| ||A^{−1}|| = \frac{σ_{max}}{σ_{min}}  (A.463)

We see that κA is large if its σ_{max} and σ_{min} are far apart in values, but it is small otherwise. It is therefore a measurement of how close A is to singularity. If A is singular, one or more of its singular values are zero, i.e., σ_{min} = 0, then κ(A) = ∞, and the system Ax = y is worst conditioned. On the other hand, if A = I, i.e., all singular values equal to 1, then κ(I) = 1, the smallest condition number possible, and the system Ix = y is best conditioned.

In Matlab, the function cond(A,p) generates the condition number of matrix A based on its p-norm. When p = 2, this is the ratio between the greatest and smallest singular values. The following Matlab commands give the same result:

• cond(A,2)
• norm(A,2)*norm(inv(A),2)
• s=svd(A), s(1)/s(end)

We now further consider the worst case scenarios for both evaluating the matrix-vector product y = Ax and solving the linear equation system Ax = y.

Consider the SVD decomposition A = UΣV^{∗} and assume the singular values in Σ are sorted in descending order: σ_{1} ≥ ⋯ ≥ σ_{n}. The SVD of the inverse is A^{−1} = VΣ^{−1}U^{∗}.

• The worst case for evaluating Ax = y:

If x = v_{n} is the minimum right singular vector corresponding to the smallest singular value σ_{n} = σ_{min}. The equation becomes:

y = Ax = (UΣV^{∗}) v_{n} = UΣ [ v_{1}^{∗} ; ⋮ ; v_{n}^{∗} ] v_{n} = U [ \begin{matrix}σ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & σ_{n} \end{matrix} ] [ 0 ⋮ 0 1 ]

= [u_{1}, ⋯, u_{n}] [ 0 ; ⋮ ; 0 ; σ_{n} ] = σ_{n}u_{n}  (A.464)

and

y + δy = A(x + δx) = σ_{n}u_{n} + Aδx  (A.465)

The absolute error ||δy|| = ||Aδx|| may be small if δx is small, but the relative error ||δy||/||y|| may be large as ||y|| = |σ_{n}| ||u|| is small when σ_{n} = σ_{min} is small (e.g., close to zero).

• The worst case for solving y = Ay:

If y = u_{1} is the maximum left singular vector corresponding to the largest singular value σ_{1} = σ_{max}. The equation becomes:

x = A^{−1}y = VΣ^{−1}U^{∗}u_{1} = VΣ^{−1} [ u_{1}^{∗} ; ⋮ ; u_{n}^{∗} ] u_{1} = V [ \begin{matrix}1/σ_{1} & ⋯ & 0 \\ ⋮ & ... & ⋮ \\ 0 & ⋯ & 1/σ_{n} \end{matrix} ] [ 1 0 ⋮ 0 ]

= [v_{1}, ⋯, v_{n}] [ 1/σ_{1} ; 0 ; ⋮ ; 0 ] = \frac{1}{σ_{1}} v_{1}  (A.466)

and

x + δx = A^{−1}(y + δy) = \frac{1}{σ_{1}} v_{1} + A^{−1}δy  (A.467)

The absolute error ||δx|| = ||A^{−1}δx|| may be small if δy is small, but the relative error ||δx||/||x|| may be large as ||x|| = ||u||/|σ_{1}| is small when σ_{1} = σ_{max} is large.

Example A.10

A = [ \begin{matrix}5.4321 & 1.2345 \\ 10.8643 & 2.4689 \end{matrix} ] ,  (A.468)

The singular values are σ_{1} = 12.45633, σ_{2} = 0.00005, and

||A|| = σ_{1} = 12.45633, ||A^{−1}|| = 1/σ_{1} = 18684.68
κ(A) = ||A|| ||A^{−1}|| = 12.45633 × 18684.68 = 232742.6

As the condition number is large, indicating A is near singular, and Ax = y is an ill-conditioned problem. To see this, consider how δy is affected by δx while assuming δA = 0:

y + δy = A^{−1}(x + δx), δy = A^{−1}δx  (A.469)

The change in output y is

||δy|| ≤ ||A^{−1}|| ||δx|| = 18684.68 ||δx||  (A.470)

which is 18684.68 times greater than any change δx in the input. However, for an evaluation problem to find y = Ax given x,

y + δy = A(x + δx), δy = Aδx  (A.471)

The change in output y is

||δy ≤ ||A|| ||δx|| = 12.45633 ||δx||  (A.472)

which is not too much greater than the change δx in the input.

Example A.11 Consider solving the linear system Ax = b with

A = \frac{1}{2} [ \begin{matrix}3.0 & 2.0 \\ 1.0 & 4.0 \end{matrix} ] , A^{−1} = [ \begin{matrix}0.8 & −0.4 \\ −0.2 & 0.6 \end{matrix} ]  (A.473)

The singular values of A are σ_{1} = 2.5583 and σ_{2} = 0.9772, the condition number is

κ = ||A|| ||A^{−1}|| = \frac{σ_{1}}{σ_{2}} = 2.618  (A.474)

which is small, indicating this is a well-behaved system. Given two similar inputs b_{1} = [1, 1]^{T} and b_{2} = [0.99, 1.01]^{T} with δb = [0.01, −0.01]^{T}, we find the corresponding solutions:

x_{1} = A^{−1}b_{1} = [ 0.4//0.4 ] , x_{2} = A^{−1}b_{2} = [ \begin{matrix}0.388 \\ 0.408 \end{matrix} ] , δx = [ \begin{matrix}0.012 \\ −0.008 \end{matrix} ]

(A.475)

We have

||δb|| = 0.0141, ||b_{1}|| = 1.4142, ||δx|| = 0.0144, ||x_{1}|| = 0.5657 (A.476)

and

\frac{||δx||/||x_{1}||}{||δb||/||b_{1}||} = \frac{0.0255}{0.01} = 2.5495  (A.477)

Example A.12

A = \frac{1}{2} [ \begin{matrix}1.000 & 1.000 \\ 1.001 & 0.999 \end{matrix} ] , A^{−1} = [ \begin{matrix}−999 & 1000 \\ 1001 & −1000 \end{matrix} ]  (A.478)

The singular values of A are σ_{1} = 1.0 and σ_{2} = 0.0005, the condition number is

κ = ||A|| ||A^{−1}|| = \frac{σ_{1}}{σ_{2}} = 2000  (A.479)

indicating matrix A is close to singularity, and the system is ill-conditioned. The solutions corresponding to the same two inputs b_{1} = [1, 1]^{T} and b_{2} = [0.99, 1.01]^{T} are

x_{1} = A^{−1}b_{1} = [ \begin{matrix}1 \\ 1 \end{matrix} ] , x_{2} = A^{−1}b_{2} = [ \begin{matrix}20.99 \\ −19.01 \end{matrix} ] , δx = [ \begin{matrix}−19.99 \\ 20.01 \end{matrix} ]

(A.480)

We can further find

||δb|| = 0.0141, ||b_{1}|| = 1.4142, ||δx|| = 28.2843, ||x_{1}|| = 1.4142 (A.481)

and the condition number is:

\frac{||δx||/||x_{1}||}{||δb||/||b_{1}||} = \frac{200}{0.01} = 2000  (A.482)

We see that a small relative change ||δb||/||b_{1}|| = 0.01 in the input caused a huge change ||δx||/||x_{1}|| = 20 in the output (2000 times greater).

Example A.13 Consider

A = [ \begin{matrix}3.1 & 2.0 \\ 6.0 & 3.9 \end{matrix} ] = UΣV^{∗} = [ \begin{matrix}−0.458 & −0.889 \\ −0.889 & 0.458 \end{matrix} ] [ \begin{matrix}8.051 & 0.0 \\ 0.0 & 0.011 \end{matrix} ] [ \begin{matrix}−0.839 & −0.544 \\ −0.544 & 0.839 \end{matrix} ]

(A.483)

with

||A|| = σ_{1} = 8.05, ||A^{−1}|| = 1/σ_{2} = 89.46, κ = ||A|| ||A^{−1}|| = 720.22

(A.484)

As A is near singular, its condition number κ is large, indicating that the problem associated with this A is ill-conditioned. We will now specifically consider both problems of evaluating and solving the linear system.

• Solve Ay = x for y given x:

y + δy = A^{−1}(x + δx), δy = A^{−1}δx  (A.485)

The absolute condition number is κ̂ = ||A^{−1}|| = 89.46. The change in output y is

||δy|| ≤ ||A^{−1}|| ||δx|| = κ̂ ||δx|| = 89.46 ||δx||  (A.486)

which is κ̂ = 89.46 times greater than any change δx in the input. We further consider the worst case when x = u_{max} and the relative error:

x = u_{1} = [ \begin{matrix}−0.458 \\ −0.889 \end{matrix} ] , y = A^{−1}x = [ \begin{matrix}−0.104 \\ −0.068 \end{matrix} ]  (A.487)

Even for a small change δx in the input, causing a small change δy in the output:

δx = [ \begin{matrix}0.01 \\ −0.01 \end{matrix} ] , δy = A^{−1}δx = [ \begin{matrix}0.656 \\ −1.011 \end{matrix} ]  (A.488)

the ratios of the absolute and relative errors are:

\frac{||δy||}{||δx||} = \frac{1.205}{0.014} ≈ 85.21,  \frac{||δy||/||y||}{||δx||/||x||} = \frac{9.702}{0.014} ≈ 686  (A.489)

• Evaluate y = Ax given x:

y + δy = A(x + δx), δy = Aδx  (A.490)

The absolute condition number is κ̂ = ||A|| = 8.05. The absolute change in output y is

||δy|| ≤ ||A|| ||δx|| = 8.051 ||δx||  (A.491)

which is not too much greater than the absolute change δx in the input. However, a much greater relative error will result if we consider the worst case when x = v_{min}:

x = v_{2} = [ \begin{matrix}−0.5444 \\ 0.8388 \end{matrix} ] , y = Ax = [ \begin{matrix}−0.0099 \\ 0.0051 \end{matrix} ]  (A.492)

Even for a small change δx in the input, causing a small change δy in the output:

δx = [ \begin{matrix}0.01 \\ −0.01 \end{matrix} ] , δy = Aδx = [ \begin{matrix}0.011 \\ 0.021 \end{matrix} ]  (A.493)

The ratios of the absolute and relative errors are:

\frac{||δy||}{||δx||} ≈ 1.68, \frac{||δy||/||y||}{||δx||/||x||} = \frac{2.1207}{0.0141} ≈ 150  (A.494)

Although the ratio of the absolute errors is small, the ratio of the relative errors is large.

A.8.6 Some Useful Inequalities

• The interpolation inequality for e^{x}

e^{ta+(1−t)b} ≤ te^{a} + (1 − t)e^{b}, t ∈ [0, 1], a, b ≥ 0  (A.495)

Proof: Evaluating the exponential function y = f(x) = e^{x} at two points x = a and x = b we get two points in the x-y plane (a, e^{a}) and (b, e^{b}). The parametric expression of the secant line through these two points is

(ta + (1 − t)b, te^{a} + (1 − t)e^{b}), (0 ≤ t ≤ 1)  (A.496)

The parametric expression of the function y = e^{x} between these two points is:

(ta + (1 − t)b, e^{ta+(1−t)b}), (0 ≤ t ≤ 1)  (A.497)

As the y = e^{x} is convex downward (or concave upward), it is never higher than the secant line function in the interval a ≤ x ≤ b, and we have

e^{ta+(1−t)b} ≤ te^{a} + (1 − t)e^{b}  (A.498)

The equality holds only when t = 0 or t = 1.

• Young’s inequality For any two real positive values p, q > 0 satisfying

\frac{1}{p} + \frac{1}{q} = 1, q = \frac{p}{p − 1}, p = \frac{q}{q − 1}  (A.499)

and any two real numbers α and β, we have

αβ ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q}  (A.500)

Proof: Substituting a = p log α and b = q log β into the interpolation inequality for f(x) = e^{x} above, we get:

e^{tp log α+(1−t)q log β} = e^{tp log α}e^{(1−t)q log β} = α^{tp}β^{(1−t)q}
≤ te^{p log α} + (1 − t)e^{q log β} = tα^{p} + (1 − t)β^{q}

We further let t = 1/p, and get 1 − t = 1 − 1/p = 1/q, the above becomes Young’s inequality:

αβ ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q}  (A.501)

• Holder’s inequality

| ∑_{i} x_{i}y_{i}| ≤ ∑_{i} |x_{i}y_{i}| ≤ ||x||_{p}||y||_{q}  (A.502)

Proof: Applying Young’s inequality to the following

α = \frac{|x_{i}|}{||x||_{p}} = \frac{|x_{i}|}{(∑_{i} |x_{i}|^{p})^{1/p}} , β = \frac{|y_{i}|}{||y||_{q}} = \frac{|y_{i}|}{(∑_{i} |y_{i}|^{q})^{1/q}}  (A.503)

we get

αβ = \frac{|x_{i}| |y_{i}|}{||x||_{p} ||y||_{q}} ≤ \frac{α^{p}}{p} + \frac{β^{q}}{q} = \frac{1}{p} \frac{|x_{i}|^{p}}{∑_{i} |x_{i}|^{p}} + \frac{1}{q} \frac{|y_{i}|^{q}}{∑_{i} |y_{i}|^{q}}  (A.504)

Taking summation on both sides we get

\frac{∑_{i} |x_{i}| |y_{i}|}{||x||_{p} ||y||_{q}} ≤ \frac{1}{p} + \frac{1}{q} = 1  (A.505)

i.e.,

∑_{i} |x_{i}y_{i}| ≤ ∑_{i} |x_{i}| |y_{i}| ≤ ||x||_{p} ||y||_{q}  (A.506)

In particular when p = 2, q = p/(p − 1) = 2, we have

| < x, y > | = | ∑_{i} x_{i}\overline{y}_{i}| = | ∑_{i} x_{i}y_{i}| ≤ ∑_{i} |x_{i}y_{i}| ≤ ||x||_{2} ||y||_{2} (A.507)

This is the Cauchy-Schwarz inequality.

• Minkowski’s inequality

||x + y||_{p} ≤ ||x||_{p} + ||y||_{p}, p ≥ 1  (A.508)

Proof:

||x + y||_{p}^{p} = ∑_{i} |x_{i} + y_{i}|^{p} = ∑_{i} |x_{i} + y_{i}| |x_{i} + y_{i}|^{p−1} ≤ ∑_{i}(|x_{i}| + |y_{i}|) |x_{i} + y_{i}|^{p−1}
= ∑_{i} |x_{i}| |x_{i} + y_{i}|^{p−1} + ∑_{i} |y_{i}| |x_{i} + y_{i}|^{p−1}

Applying Holder’s inequality to each of the two terms on the right-hand side, we get

||x + y||_{p}^{p} ≤ ||x||_{p} (∑_{i} |x_{i} + y_{i}|^{(p−1)q})^{1/q} + ||y||_{p} (∑_{i} |x_{i} + y_{i}|^{(p−1)q})^{1/q}

But as 1/p + 1/q = 1, and

q = p/(p − 1), 1/q = (p − 1)/p, (p − 1)q = p  (A.510)

the above becomes:

||x+y||_{p}^{p} ≤ ||x||_{p} (∑_{i} |x_{i} + y_{i}|^{p})^{(p−1)/p}+||y||_{p} (∑_{i} |x_{i} + y_{i}|^{p})^{(p−1)/p} = (||x||_{p} + ||y||_{p}) ||x+y||_{p}^{p−1}

(A.511)

Multiplying both sides by ||x + y||_{p}^{1−p}, we get Minkowski’s inequality.

Appendix B A Review of Probability and Statistics

B.1 Univariate and Multivariate Random Variables

B.1.1 Random Events and Probability Space

• Random Experiment and Sample Space

A random experiment is a procedure that can be repeated any number of times and has a set of possible outcomes. The sample space S of a certain random experiment is a set of all its possible outcomes, containing either a finite number of discrete (countable) outcomes, or infinite continuous (uncountable) outcomes.

For example, the random experiment “Randomly pick a card from a deck of 10 cards labeled 0, 1, ⋯, 9” has a discrete sample space, a set of the 10 possible outcomes: S = {0, 1, ⋯, 9}.

• Random Event and Probability Space

A random event A ⊂ S is a subset of S, which can be a null or empty set ∅, a proper subset, e.g., A = {0, 2, 4, 6, 8} ⊂ S = {0, 1, ⋯, 9}, or the entire S. Event A occurs if the outcome s ∈ A.

The probability P (A) of an event A is a real-valued function that maps A to a real number 0 ≤ P (A) ≤ 1. Specially, P (∅) = 0, and P (S) = 1.

For example, the random event “The randomly chosen card has a number smaller than 4” is represented by a subset A = {0, 1, 2, 3} ⊂ S, and its probability is P (A) = 4/10 = 0.4. Event A occurs if the outcome is one of the member of A (e.g., 2).

The triple (S, A, P) of sample space S, events A and probability P is called the probability space.

• Intersection, Union, and Complement

The Intersection A ∪ B of events A and B is an event whose outcomes belong to both A and B. If A and B are mutually exclusive, i.e., A∩B = ∅, then P (A ∩ B) = 0.

The Union (OR) A ∪ B of events A and B is an event whose outcomes belong to either A or B

P (A ∪ B) = P (A) + P (B) − P (A ∩ B)  (B.1)

If A and B are mutually exclusive, then

P (A ∪ B) = P (A) + P (B)  (B.2)

The complement A^{C} of an event A is a set of all outcomes not in A. Obviously A ∪ A^{C} = S, A ∩ A^{C} = ∅, and P (A^{C}) = 1 − P (A).

• Conditional Probability and Independence

The conditional probability P (A|B) is the probability of event A given that event B has already occurred, which can be expressed as

P (A ∩ B) = P (B|A) P (A) = P (A|B) P (B)  (B.3)

which can also expressed in terms of the conditional probability of A given B:

P (A|B) = \frac{P (A ∩ B)}{P (B)} = \frac{P (B|A)P (A)}{P (B)}  (B.4)

Two events A and B are independent if and only if

P (A ∩ B) = P (A)P (B)  (B.5)

i.e.,

P (A|B) = \frac{P (A, B)}{P (B)} = \frac{P (A)P (B)}{P (B)} = P (A)  (B.6)

We further have

\begin{matrix}P (A|B) P (B|C) = P (B|A) P (A|C) \\ P (A|B, C) P (B|C) = P (B|A, C) P (A|C) \end{matrix} \} = P (A ∩ B|C) (B.7)

• Partition Theorem

Let {A_{1}, ⋯, A_{n}} be a set of mutually exclusive events that partition the sample space S, i.e.,

⋃_{i=1}^{n} A_{i} = S, and A_{i} ∩ A_{j} = ∅ ∀i ≠ j  (B.8)

then for any event B ⊂ S, we have

P (B) = ∑_{i=1}^{n} P (B ∩ A_{i}) = ∑_{i=1}^{n} P (B|A_{i})P (A_{i})  (B.9)

Proof:

P (B) = P (B ∩ S) = P (B ∩ (⋃_{i=1}^{n} A_{i})) = P (⋃_{i=1}^{n}(B ∩ A_{i}))

= ∑_{i=1}^{n} P (B ∩ A_{i}) = ∑_{i=1}^{n} P (B|A_{i})P (A_{i})  (B.10)

We further have:

P (A_{i}|B) = \frac{P (B|A_{i}) P (A_{i})}{P (B)} = \frac{P (B|A_{i}) P (A_{i})}{∑_{i=1}^{n} P (B|A_{i})P (A_{i})}  (B.11)

• Bayes’ Theorem

Bayes’ theorem is based on the expression for the conditional probability of A given B in Eq. (B.4):

p(A|B) = \frac{p(A, B)}{p(B)} = \frac{p(B|A) p(A)}{p(B)}  (B.12)

If {A_{1}, ⋯, A_{n}} is a partition of sample space A, then

P (B) = ∑_{i=1}^{n} P (B|A_{i}) P (A_{i})
P (A_{i}|B) = \frac{P (B|A_{i}) P (A_{i})}{P (B)} = \frac{P (B|A_{i}) P (A_{i})}{∑_{i=1}^{n} P (B|A_{i}) P (A_{i})} (i = 1, ⋯, n) (B.13)

Here the denominator P (B) = ∑_{i=1}^{n} P (B|A_{i}) P (A_{i}) can be considered as a normalizing factor by which P (A_{i}|B) is normalized to satisfy ∑_{i=1}^{n} P (A_{i}|B) = 1.

For example, let P (A_{i}) be the probability for a student to take any one of some n elective courses {A_{1}, ⋯, A_{n}}, and P (B|A_{i}) be the probability for the student to get a B grade in course A_{i}. Given that the student gets a B, the probability for him/her to have taken course A_{i} is P (A_{i}|B).

B.1.2 Univariate Random Variables

• Random Variable

A random variable x = x(s) is a real-valued function that maps any outcome s ∈ S in the sample space into a real number x, which is either discrete if S is countable or continuous if S is uncountable.

• Cumulative and Density Distribution Functions

The probability mass function (pmf) of a discrete random variable x with n possible vales {x_{1}, ⋯, x_{n}} is:

P (x = x_{i}) = P (x_{i}) = P_{i} (i = 1, ⋯, n)  (B.14)

satisfying

0 ≤ P_{i} ≤ 1,  ∑_{i=1}^{n} P_{i} = 1  (B.15)

The cumulative distribution function is

F_{x}(ξ) = P (x < ξ) = ∑_{x_{i}<ξ} P (x_{i}) = ∑_{i=1}^{k} P_{i}  (B.16)

where we assumed x_{1} < ⋯ < x_{n} and ξ = x_{k+1}.

The cumulative distribution function (cdf) of a continuous random variable x is defined as:

F_{x}(ξ) = P (x < ξ)  (B.17)

The probability density function (pdf) of x is

p(x) = \frac{d}{dξ} F_{x}(ξ), i.e. F_{x}(ξ) = ∫_{−∞}^{ξ} p(x)dx  (B.18)

and we have

P (a ≤ x < b) = F_{x}(b) − F_{x}(a) = ∫_{a}^{b} p(x)dx  (B.19)

and

F_{x}(∞) = ∫_{−∞}^{∞} p(x)dx = 1  (B.20)

A random variable x with pdf p(x) is denoted by x ∼ p(x). Sometimes we may also denote a random variable by X and its pdf by p_{X}(x) for clarity, such as to distinguish the different pdfs of two variables, X ∼ p_{X}(x) and Y ∼ p_{Y} (y).

• Expectation

The expectation value (mean or average) of a random variable x is the average of all possible values it can take weighted by their corresponding probabilities:

μ = E[x] = \{ \begin{matrix}∑_{i=1}^{n} x_{i}P_{i} & x is discrete \\ ∫_{−∞}^{∞} x p(x)dx & x is continuous \end{matrix}  (B.21)

• Variance The variance of a random variable x is

σ^{2} = Var[x] = E[(x − μ)^{2}] = \{ \begin{matrix}∑_{i=1}^{n}(x_{i} − μ)^{2}P_{i} & x is discrete \\ ∫_{−∞}^{∞}(x − μ)^{2}p(x)dx & x is continuous \end{matrix}

(B.22)

The variance can also be written as:

σ^{2} = Var[x] = E[(x − μ)^{2}] = E[x^{2} − 2xμ + μ^{2}]
= E[x^{2}] − 2μ_{x} E[x] + μ^{2} = E[x^{2}] − 2μ^{2} + μ^{2} = E[x^{2}] − μ^{2} (B.23)

The standard deviation of x is defined as

\sqrt{Var[x]} = \sqrt{σ^{2}} = σ  (B.24)

The variance σ_{x}^{2} ≥ 0 is non-negative and it measures the dispersion of the values of variable x, i.e., how widely or narrowly its values distribute over the real axis. Specially if σ_{x}^{2} = 0, then x = μ_{x} is no longer a random variable.

• Estimation of Mean and Variance When the pmf P_{i} or pdf p(x) of random variable x is unknown, its mean μ and variance σ^{2} can be estimated based on a set of independent samples {x_{1}, ⋯, x_{N}} of x:

μ̂ = \frac{1}{N} ∑_{n=1}^{N} x_{n}  (B.25)

σ̂^{2} = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} − μ̂)^{2}  (B.26)

Note that while estimating σ^{2}, the sum is divided by N −1 instead of N, so that the estimated variance is unbiased, i.e., E[σ̂^{2}] = σ^{2}, as shown below:

E[σ̂^{2}] = \frac{1}{N − 1} E [∑_{n=1}^{N}(x_{n} − μ̂)^{2}] = \frac{1}{N − 1} E [∑_{n=1}^{N}[(x_{n} − μ) − (μ̂ − μ)]^{2}]

= \frac{1}{N − 1} E [∑_{n=1}^{N}(x_{n} − μ)^{2} − 2 ∑_{n=1}^{N}(x_{n} − μ)(μ̂ − μ) + ∑_{n=1}^{N}(μ̂ − μ)^{2}]

=^{1} \frac{1}{N − 1} E [∑_{n=1}^{N}(x_{n} − μ)^{2}] − \frac{N}{N − 1} E [(μ̂ − μ)^{2}]
=^{2} \frac{N}{N − 1}σ^{2} − \frac{1}{N − 1}σ^{2} = σ^{2}  (B.27)

where the labeled equal signs are due to the following reasons: 1.The sum in the second term can written as

∑_{n=1}^{N}(x_{n} − μ) = N(μ̂ − μ)  (B.28)

2.The expectation in the second term is the variance of μ̂ (Eq. (B.81)):

E[(μ̂ − μ)^{2}] = Var[μ̂] = σ^{2}/N  (B.29)

• Standardization

A given random variable x with mean μ_{x} and variance σ_{x}^{2} can be demeaned or centered by removing its mean:

y = x − μ_{x}  (B.30)

so that the resulting variable y has a zero mean

μ_{y} = E[y] = E[x − μ_{x}] = μ_{x} − μ_{x} = 0  (B.31)

but the same variance σ_{y}^{2} = σ_{x}^{2}. This conversion is often carried out for mathematical convenience without loss of generality.

Moreover, variable y obtained above can be further scaled by 1/σ_{x} to become

z = \frac{y}{σ_{x}} = \frac{x − μ_{x}}{σ_{x}}  (B.32)

so that the resulting variable z not only has a zero mean μ_{z} = 0 but also a unit variance:

σ_{z}^{2} = Var[z] = E [\frac{(x − μ_{x})^{2}}{σ_{x}^{2}} ] = \frac{E[(x − μ_{x})^{2}]}{σ_{x}^{2}} = \frac{σ_{x}^{2}}{σ_{x}^{2}} = 1  (B.33)

The process combining these two steps to convert x to z is called standardization.

• The Median and Mode

The median x̃ of a random variable x is defined as the middle value of its distribution. If x is discrete, then

x̃ = \{ \begin{matrix}x_{(n+1)/2} & if n is odd \\ (x_{n/2} + x_{n/2+1})/2 & if n is even \end{matrix}  (B.34)

If x is continuous, then the median x̃ is defined such that

∫ x̃ ; −∞ p(x)dx = ∫ ∞ ; x̃ p(x)dx = 0.5  (B.35)

The mode of variable x is the value that appears most frequently among all possible values:

x̃ = arg max_{x} p(x), i.e., p(x̃) = max_{x} p(x)  (B.36)

For example, given n = 8 values in {1, 2, 3, 3, 4, 4, 4, 5}, we can find their mean as

x̄ = \frac{1}{8}(1+2+3+3+4+4+4+5) = 1×\frac{1}{8}+2×\frac{1}{8}+3×\frac{2}{8}+4×\frac{3}{8}+5×\frac{1}{8} = \frac{26}{8} = 3.25

(B.37)

The median and mode are respectively 3.5 and 4.

• Joint Probability and Marginalization The joint probability of two random variables X and Y is

P (x, y) = P ((X = x) ∩ (Y = y)) = P (x|y)P (y) = P (y|x)P (x) (B.38)

where P (x|y) is the conditional probability of event X = x given that Y = x. This definition can be generalized to the joint probability P (x_{1}, ⋯, x_{n}) of multiple variables. If these variables are independent, then

P (x_{1}, ⋯, x_{n}) = ∏_{i=1}^{n} P (x_{n})  (B.39)

For continuous variables, we have

p(x, y) = p(x|y) p(y) = p(y|x) p(x)  (B.40)

If they are independent, then we have

p(x|y) = p(x), p(y|x) = p(y), p(x, y) = p(x) p(y)  (B.41)

Given the joint probability p(x, y), we can find the probability p(x) by marginalizing variable y:

p(x) = ∫ p(x, y) dy = ∫ p(x|y)p(y) dy = E_{y}[p(x|y)]  (B.42)

B.1.3 Multivariate Random Vectors

• Multiple Random Variables

An outcome of a random experiment may need to be described by a set of n > 1 random variables {x_{1}, ⋯, x_{n}}, which can be more concisely represented as a random vector x = [x_{1}, ⋯, x_{n}]^{T}. For example, in signal processing x can be a set of time samples x_{i} = x(t_{i}) of a random signal x(t) also called a stochastic process, and in pattern recognition x can be a pattern representing an object of interest, in terms of its n features {x_{1}, ⋯, x_{n}} that describe the object.

• Joint Distribution Function and Density Function The joint distribution function of a random vector x is defined as

F_{x}(u_{1}, ⋯, u_{n}) = P (x_{1} < u_{1}, ⋯, x_{n} < u_{n})
= ∫_{−∞}^{u_{1}} ⋯ ∫_{−∞}^{u_{n}} p(ξ_{1}, ⋯, ξ_{n})dξ_{1} ⋯ dξ_{n} (B.43)

where p(ξ_{1}, ⋯, ξ_{n}) is the joint density function of the random vector x in the n-dimensional vector space. In particular, if all these random variables are independent, then p(x_{1}, ⋯, x_{n}) = p(x_{1}) ⋯ p(x_{n}) = ∏_{i=1}^{n} p(x_{i}).

• Mean Vector The expectation or mean of a random variable x_{i} is defined as

μ_{i} = E[x_{i}] = ∫_{−∞}^{∞} ⋯ ∫_{−∞}^{∞} ξ_{i} p(ξ_{1}, ⋯, ξ_{n}) dξ_{1} ⋯ dξ_{n}  (B.44)

The mean vector of a random vector x is defined as

m = E[x] = [E[x_{1}], ⋯, E[x_{n}]]^{T} = [μ_{1}, ⋯, μ_{n}]^{T}  (B.45)

which can be interpreted as the center of gravity of an n-dimensional object with p(x_{1}, ⋯, x_{n}) being the density function.

• Covariance Matrix The covariance of any two variables x_{i} and x_{j} is defined as

σ_{ij}^{2} = Cov[x_{i}, x_{j}] = E[(x_{i} − μ_{i})(x_{j} − μ_{j})]
= E[x_{i}x_{j}] − E[x_{i}]μ_{j} − μ_{i} E[x_{j}] + μ_{i}μ_{j} = E[x_{i}x_{j}] − μ_{i}μ_{j}
= ∫_{−∞}^{∞} ∫_{−∞}^{∞} ξ_{i}ξ_{j} p(ξ_{i}, ξ_{j}) dξ_{i} dξ_{j} − μ_{i}μ_{j}  (B.46)

where E[x_{i}x_{j}] = r_{ij} = σ_{ij}^{2} + μ_{i}μ_{j} is the correlation of variables x_{i} and x_{j}. Note that σ_{ij}^{2} = σ_{ji}^{2} is either positive or negative, and so is r_{ij}. But the variance σ_{ii}^{2} = E[(x_{i} −μ_{i})^{2}] ≥ 0 is nonnegative. It can be interpreted as the amount of dynamic energy or information contained in x_{i} as a sample of a signal represented by x.

If σ_{ij}^{2} = 0, then variables x_{i} and x_{j} said to be uncorrelated. If x_{i} and x_{j} are independent, i.e., p(x_{i}, x_{j}) = p(x_{i})p(x_{j}), then E[x_{i}x_{j}] = μ_{i}μ_{j}. and the covariance defined above is σ_{ij}^{2} = E[x_{i}] E[x_{j}] − μ_{i}μ_{j} = 0, i.e., x_{i} and x_{j} are uncorrelated. However, two uncorrelated variables are not necessarily independent. If x is normally distributed, then its components are independent if and only they are uncorrelated.

The covariance matrix of a random vector x is defined as

Σ_{x} = E[(x − m) (x − m)^{T}] = E[xx^{T}] − mm^{T} = R − mm^{T}
= [ \begin{matrix}σ_{11}^{2} & ⋯ & σ_{1n}^{2} \\ ⋮ & ... & ⋮ \\ σ_{n1}^{2} & ⋯ & σ_{nn}^{2} \end{matrix} ]  (B.47)

where R = E[xx^{T}] = Σ_{x} + mm^{T} is the correlation matrix:

R_{x} = E[xx^{T}] = [ \begin{matrix}E[x_{1}^{2}] & ⋯ & E[x_{1}x_{n}] \\ ⋮ & ... & ⋮ \\ E[x_{n}x_{1}] & ⋯ & E[x_{n}^{2}] \end{matrix} ] , = [ \begin{matrix}r_{11}^{2} & ⋯ & r_{1n}^{2} \\ ⋮ & ... & ⋮ \\ r_{n1}^{2} & ⋯ & r_{nn}^{2} \end{matrix} ]

(B.48)

If m = 0, then Σ = R.

The trace of Σ_{x} as the sum of all n variances of x_{i} can be interpreted as the total dynamic energy or information contained in x:

tr Σ = ∑_{i=1}^{n} σ_{i}^{2} = ∑_{i=1}^{n} λ_{i}  (B.49)

where λ_{1}, ⋯, λ_{n} are the real and non-negative eigenvalues of the positive semi-definite matrix Σ.

Specially if σ_{ij} = 0 for all i ≠ j, the random vector x is uncorrelated, and the covariance matrix becomes diagonal Σ = diag(σ_{1}^{2}, ⋯, σ_{n}^{2}) with λ_{i} = σ_{i}^{2}.

• Correlation Coefficient

The Pearson correlation coefficient between two random variables x_{i} and x_{j} is the normalized covariance σ_{ij}^{2}, when it is divided by the standard deviations σ_{i} and σ_{j}:

ρ_{ij} = \frac{σ_{ij}^{2}}{\sqrt{σ_{i}^{2}σ_{j}^{2}}} = \frac{σ_{ij}^{2}}{σ_{i}σ_{j}}  (B.50)

As the covariance can be considered as an inner product of x_{i} and x_{j}

σ_{ij}^{2} = E[(x − μ_{i}) (x − μ_{j})] = ⟨x_{i}, x_{j}⟩  (B.51)

the Cauchy-Schwarz inequality (Eq. (A.23)) applies:

|⟨x_{i}, x_{j}⟩|^{2} = σ_{ij}^{4} ≤ ⟨x_{i}, x_{i}⟩⟨x_{j}, x_{j}⟩ = σ_{i}^{2}σ_{j}^{2}  (B.52)

i.e.,

(\frac{σ_{ij}^{2}}{σ_{i}σ_{j}} )^{2} = ρ^{2} ≤ 1,  −1 ≤ ρ ≤ 1  (B.53)

The correlation coefficient of two random variables x and y can estimated by the sample correlation coefficient based on a set of samples of the two variables {(x_{i}, y_{i}), i = 1, ⋯, k}:

ρ̂_{xy} = \begin{matrix} & σ̂_{xy}^{2} \\ σ̂_{x} & σ̂_{y} \end{matrix} = \frac{∑_{i=1}^{k}(x_{i} − μ_{x})(y_{i} − μ_{y})}{\sqrt{(∑_{i=1}^{k}(x_{i} − μ_{x})^{2}) (∑_{i=1}^{k}(y_{i} − μ_{y})^{2})}}  (B.54)

• Mean and Covariance under Orthogonal Transforms

Given an orthogonal matrix A = [a_{1}, ⋯, a_{n}] composed of n orthogonal column vectors satisfying a_{i}^{T} a_{j} = δ_{ij}, i.e., A^{T} = A^{−1}, an orthogonal transform of x can be defined as

y = [ y_{1} ; ⋮ ; y_{n} ] = A^{T} x = [ a_{1}^{T} ; ⋮ ; a_{n}^{T} ] x, i.e. y_{i} = a_{i}^{T} x, (i = 1, ⋯, n)

(B.55)

The resulting y is also a random vector. The inverse transform is:

x = Ay = [a_{1}, ⋯, a_{n}] [ y_{1} ; ⋮ ; y_{n} ] = ∑_{i=1}^{n} y_{i}a_{i}  (B.56)

Consider the squared norm of vector x, the total amount of energy contained in x:

||x||^{2} = x^{T} x = (∑_{i=1}^{n} y_{i}a_{i})^{T} (∑_{j=1}^{n} y_{j}a_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} y_{i}y_{j} a_{i}^{T} a_{j} = ∑_{i=1}^{n} y_{i}^{2} = ||y||^{2}

(B.57)

This is Parseval’s theorem, indicating that the total energy is conserved under any orthogonal transform.

The mean vector m_{y} and the covariance matrix Σ_{y} of y are related to the m_{x} and Σ_{x} of x by:

m_{y} = E[y] = E[A^{T} x] = A^{T} E[x] = A^{T} m_{x}  (B.58)

and

Σ_{y} = E[yy^{T}] − m_{y}m_{y}^{T} = E[A^{T} xx^{T} A] − A^{T} m_{x}m_{x}^{T} A
= A^{T} E[xx^{T}]A − A^{T} m_{x}m_{x}^{T} A = A^{T} [E[xx^{T}] − m_{x}m_{x}^{T}]A = A^{T} Σ_{x}A (B.59)

The orthogonal transform does not change the trace of Σ: Due to the commutativity of trace: tr(AB) = tr(BA), we have:

tr Σ_{y} = tr(A^{T} Σ_{x}A) = tr(A^{T} AΣ_{x}) = tr Σ_{x}  (B.60)

i.e., the total amount of dynamic energy or information contained in x is conserved by the orthogonal transform y = A^{T} x.

• Estimation of m and Σ

When p(x) = p(x_{1}, ⋯, x_{n}) is not known, m and Σ cannot be found as defined in Eqs. (B.45) and (B.46). However, they can be estimated if a set of samples X = [x_{1}, ⋯, x_{N}] of x is available. The sample mean m̂ is:

m̂ = \frac{1}{N} ∑_{n=1}^{N} x_{n}  (B.61)

and the unbiased sample covariance is:

Σ̂ = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} − m̂)(x_{n} − m̂)^{T}  (B.62)

If all N samples in the dataset X are independent, then the rank of the n × n sample covariance matrix Σ̂ is at most n − 1, due to the constraining equation:

∑_{n=1}^{N}(x_{n} − m̂) = ∑_{n=1}^{N} x_{n} − Nm̂ = 0  (B.63)

When N < n−1, i.e., the sample size is smaller than the rank of the sample covariance matrix Σ̂, then it does not have a full rank and is not invertible.

• Standardization

Same as in the case of univariate random variables, a multivariate random vector x with mean m_{x} and covariance Σ_{x} can also be standardized. First, we remove the mean to get y = x − m_{x}, so that m_{y} = 0 and the same covariance Σ_{y} = Σ_{x}. We then consider the eigen-equation of this matrix (Eq. (A.95)):

V^{−1}Σ_{x}V = V^{T} Σ_{x}V = Λ  (B.64)

As Σ_{x} is positive semidefinite, its eigenvalues in the diagonal eigenvalue matrix Λ are non-negative, and its eigenvector matrix V^{−1} = V^{T} is orthogonal. We can now carry out another transform to get

z = (Λ^{−1/2}V^{T})y  (B.65)

with zero mean m_{z} = m_{y} = 0 and covariance

Σ_{z} = E[zz^{T}] = E [(Λ^{−1/2}V^{T} y)(Λ^{−1/2}V^{T} y)^{T} ]
= Λ^{−1/2}V^{T} E[yy^{T}]VΛ^{−1/2} = Λ^{−1/2}(V^{T} Σ_{x}V)Λ^{−1/2}
= Λ^{−1/2}ΛΛ^{−1/2} = I  (B.66)

i.e., all components of z are decorrelated and they all have zero mean and unit variance.

B.1.4 Function of Random Variables

A function Y = f(X) of a random variable X with pdf p_{X}(x) is also a random variable, and its pdf p_{Y} (y) can be obtained based on p_{X}(x) if the function is differentiable and invertible, i.e., a unique x = f^{−1}(y) exists. We first consider its cdf in the following two cases. First, if y = f(x) is a monotonically increasing function (dy/dx > 0), then its cdf is

F_{Y} (y) = P (Y < y) = P (f(X) < y) = P (X < f^{−1}(y)) = ∫_{−∞}^{x} p_{X}(u)du|_{x=f^{−1}(y)}

(B.67)

based on which we can find its pdf:

p_{Y} (y) = \frac{d}{dy} F_{Y} (y) = \frac{d}{dy} ∫_{−∞}^{x} p_{X}(u)du|_{x=f^{−1}(y)} = [p_{X}(x)\frac{dx}{dy}]_{x=f^{−1}(y)} (B.68)

Next, if y = f(x) is monotonically decreasing (dy/dx < 0), then its cdf is:

F_{Y} (y) = P (Y < y) = P (f(X) < y) = P (X > f^{−1}(y)) = ∫_{x}^{∞} p_{X}(u)du|_{x=f^{−1}(y)}

(B.69)

and its pdf is:

p_{Y} (y) = \frac{d}{dy} F_{Y} (y) = \frac{d}{dy} ∫_{x}^{∞} p_{X}(u)du|_{x=f^{−1}(y)} = [−p_{X}(x)\frac{dx}{dy}]_{x=f^{−1}(y)} (B.70)

Combining these two cases, we get

p_{Y} (y) = [p_{X}(x) |\frac{dx}{dy}|]_{x=f^{−1}(y)}  (B.71)

Alternatively, consider the probability for a ≤ x ≤ b, same as the probability for f(a) ≤ y = f(x) ≤ f(b), i.e.,

∫_{a}^{b} p_{X}(x)dx = ∫_{f(a)}^{f(b)} p_{Y} (y)dy  (B.72)

If we assume the interval b − a is infinitesimally small and so is f(b) − f(a), then p_{X}(x) and p_{Y} (y) can be assumed to be constants, we have

p_{X}(x)dx = p_{Y} (y)dy, i.e., p_{Y} (y) = [p_{X}(x) |\frac{dx}{dy}|]_{x=f^{−1}(y)}  (B.73)

where the absolute value |dx/dy| ≥ 0 guarantees that p_{Y} (y) > 0 as well as p_{X}(x) > 0.

As a simple example, consider the pdf of Y = f(X) = cX, where c is the constant and X ∼ p_{X}(x), i.e., x = f^{−1}(y) = y/c and dx/dy = 1/c. Now we have

Y = cX ∼ p_{Y} (y) = [p_{X}(x) |\frac{dx}{dy}|]_{x=f^{−1}(y)} = \frac{1}{c} p_{X} (\frac{y}{c} )  (B.74)

with mean and variance:

μ_{y} = E[Y] = E[cX] = c E[X] = c μ_{x}
σ_{y}^{2} = E[Y^{2} − μ_{y}^{2}] = E[(cX)^{2} − (cμ_{x})^{2}] = c^{2}σ_{x}^{2}  (B.75)

We next consider the mean and variance of y = x_{1} + ⋯ + x_{N} as the sum of N random variables:

μ_{y} = E[y] = E [∑_{i=1}^{N} x_{i}] = ∑_{i=1}^{n} E[x_{i}] = ∑_{i=1}^{N} μ_{x_{i}}  (B.76)

We also have

E[y^{2}] = E [(∑_{i=1}^{N} x_{i})^{2}] = E [∑_{i=1}^{N} ∑_{j=1}^{N} x_{i}x_{j}] = ∑_{i=1}^{N} ∑_{j=1}^{N} E[x_{i}x_{j}]  (B.77)

and

(E[y])^{2} = (∑_{i=1}^{N} E[x_{i}])^{2} = ∑_{i=1}^{N} ∑_{j=1}^{N} E[x_{i}] E[x_{j}]  (B.78)

then we get

σ_{y}^{2} = Var(y) = E[y^{2}] − (E[y])^{2} = ∑_{i=1}^{N} ∑_{j=1}^{N} E[x_{i}x_{j}] − ∑_{i=1}^{N} ∑_{j=1}^{N} E[x_{i}] E[x_{j}]

= ∑_{i=1}^{N} ∑_{j=1}^{N} Cov[x_{i}, x_{j}] = ∑_{i=1}^{N} Var[x_{i}] + 2 ∑_{i<j} Cov[x_{i}, x_{j}]  (B.79)

Specially if all x_{i} are uncorrelated (or more specially independent), i.e., Cov[x_{i}, x_{j}] = 0 for all i ≠ j, we have

σ_{y}^{2} = ∑_{i=1}^{N} Var[x_{i}] = ∑_{i=1}^{N} σ_{x_{i}}^{2}  (B.80)

As one more example, consider the variance of the estimated mean in Eq. (B.25) based on N i.i.d. samples:

Var[μ̂] = Var [\frac{1}{N} ∑_{n=1}^{N} x_{n}] = \frac{1}{N^{2}} ∑_{n=1}^{N} Var[x_{n}] = \frac{σ^{2}}{N}  (B.81)

We next consider a random vector x and a function

y = [y_{1}, ⋯, y_{m}]^{T} = [f_{1}(x), ⋯, f_{m}(x)]^{T} f(x)  (B.82)

If the function is differentiable and invertible, i.e., x = f^{−1}(y) exists, then we can find the pdf of y given the pdf x:

p_{Y} (y) = [p_{X}(x) |det J_{f^{−1}} |]_{x=f^{−1}(y)}  (B.83)

where | det J_{f^{−1}}| is the absolute value of the determinant of the Jacobian matrix:

J_{f^{−1}} = [ \begin{matrix}\frac{∂x_{1}}{∂y_{1}} & ⋯ & \frac{∂x_{1}}{∂y_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂x_{m}}{∂y_{1}} & ⋯ & \frac{∂x_{m}}{∂y_{n}} \end{matrix} ]_{x=f^{−1}(y)}  (B.84)

Specially, for a linear function y = f(x) = Ax, where A is an invertible square matrix, we have J_{f} = A and J_{f^{−1}} = A^{−1}, and

p_{Y} (y) = p_{Y} (Ax) = [| det A^{−1}|p_{X}(x)]_{x=A^{−1}y}  (B.85)

The mean and covariance of a function of a random vector can be approximated based on its Taylor expansion near the mean. Consider y = f(x) as a nonlinear function of a random variable x. Its Taylor expansion near the mean μ_{x} is:

y = f(x) = f(μ_{x})+f′(μ_{x})(x−μ_{x})+ \frac{f′′(μ_{x})}{2}(x−μ_{x})^{2}+⋯ ≈ f(μ_{x})+f′(μ_{x})(x−μ_{x})

(B.86)

Taking the mean on both side we get

μ_{y} = E[y] ≈ E[f(μ_{x})] + f′(μ_{x}) E[x − μ_{x}] = f(μ_{x})  (B.87)

Taking the variance on both sides we get

Var[y] ≈ Var[f(μ_{x}) + f′(μ_{x})(x − μ_{x})] = Var[f′(μ_{x})(x − μ_{x})]
= f^{′2}(μ_{x}) Var[x] = f^{′2}(μ_{x})σ_{x}^{2}  (B.88)

For a function y = f(x) of a random vector x, its Taylor expansion in the neighborhood of the mean vector m_{x} is:

f(x) = f(m_{x}) + g(m_{x})^{T} (x − m_{x}) + \frac{1}{2}(x − m_{x})^{T} H(m_{x})(x − m_{x}) + ⋯
≈ f(m_{x}) + g(m_{x})^{T} (x − m_{x})  (B.89)

where

g(m_{x}) = [ ⋮_{\frac{∂fx)}{∂x_{n}}}^{\frac{∂fx)}{∂x_{1}}} ]_{m_{x}} , H(m_{x}) = [ \begin{matrix}\frac{∂^{2}f(x)}{∂x_{1}^{2}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{1}∂x_{n}} \\ ⋮ & ... & ⋮ \\ \frac{∂^{2}f(x)}{∂x_{n}∂x_{1}} & ⋯ & \frac{∂^{2}f(x)}{∂x_{n}^{2}} \end{matrix} ]_{m_{x}} (B.90)

are the gradient vector and Hessian matrix of f(x) evaluated at m_{x}. Taking the mean of the expression above

μ_{y} ≈ Ef(x) ≈ E [f(m_{x})] g E[x − m_{x}] = f(m_{x})  (B.91)

Taking the variance we get

σ_{y}^{2} = Var [f(x)] ≈ Var [f(m_{x}) + g(m_{x})^{T} (x − m_{x})] = Var [g(m_{x})^{T} (x − m_{x})]
= Var [∑_{i=1}^{n} \frac{∂f}{∂x_{i}} (x_{i} − μ_{x_{i}})] = ∑_{i=1}^{n} (\frac{∂f}{∂x_{i}} )^{2} σ_{x_{i}}^{2}  (B.92)

or

σ_{y} = \sqrt{∑_{i=1}^{n} (\frac{∂f}{∂x_{i}} )^{2} σ_{x_{i}}^{2}}  (B.93)

B.1.5 The Normal Distribution

We first consider the central limit theorem (CLT), which states that the distribution of the sum of a number of independent and identically distributed (i.i.d.) random variables of arbitrary distribution tends to a normal distribution as the number of variables approaches infinity.

Specifically, Let {x_{1}, ⋯, x_{N}} be a set of i.i.d. samples drawn from an arbitrary distribution with mean μ and variance σ^{2}. Then the the CLT states that the distribution of the sample average

x̄_{N} = \frac{1}{N} ∑_{n=1}^{N} x_{i}  (B.94)

tends to a normal distribution p(x) = N (x, μ, σ^{2}/N).

For example, the face value of a dice has a uniform distribution P (v = 1) = ⋯ = P (v = 6) = 1/6. But the distribution of the sum of a pair of dice is no longer uniform. It has a maximum probability at the mean of 7. As the number of dice increases, the distribution of the sum of the face values approaches a Gaussian, as illustrated in the figure below showing the estimated distribution of the sum of N = 1, 2, 4 and 8 dice.

The normal or Gaussian distribution is the most important pdf widely used to model observed data set drawn from an unknown distribution, due to two important facts, first, the central limit theorem discussed above, and second, a normal distribution has the maximum entropy or uncertainty among all possible probability density functions with the same variance, which we will prove later.

The univariate normal probability distribution is

p(x) = N (x, μ, σ^{2}) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{(x − μ)^{2}}{2σ^{2}} )  (B.95)

with the following mean and variance

E[x] = μ,  Var[x] = σ^{2}  (B.96)

This is a unimodal bell-shaped curve centered at x = μ, which is the mean, median and mode of the distribution. As the curve is symmetric with respect to μ, its skewness is zero. Its excess kurtosis (Section B.2.4) is zero.

In the multivariate case, the pdf of a normally distributed n-dimensional random vector x is

p(x) = N (x, m, Σ) = \frac{1}{(2π)^{n/2} |Σ|^{1/2}} exp (−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)) (B.97)

with mean vector m = E[x] and covariance matrix Σ = Cov[x]. We further define the log-normal distribution as:

ψ(x) = log N (x, m, Σ) = −\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m) − \frac{n}{2} log(2π) − \frac{1}{2} log |Σ|

(B.98)

Its gradient vector and Hessian matrix are:

g_{ψ}(x) = \frac{d}{dx}ψ(x) = −Σ^{−1}(x − m)  (B.99)
H_{ψ}(x) = \frac{d^{2}}{dx^{2}} ψ(x) = −Σ^{−1}  (B.100)

Setting the first expression to zero and solving the resulting equation we get the mode x = m at which N (x, m, Σ) reaches maximum. Solving the second equation for Σ we get the covariance:

Σ = −H_{ψ}^{−1}(x)  (B.101)

The iso-hypersurface of the normal distribution is represented by the following equation:

N (x, m, Σ) = c_{0}  (B.102)

where c_{0} is a constant. Taking the log on both sides of the equation and ignore the constant scaling factor, we get an equivalent quadratic equation:

(x − m)^{T} Σ^{−1}(x − m) = c_{1}  (B.103)

This equation in general represents a quadratic (e.g., elliptic, parabolic, hyperbolic) hypersurface in a high dimensional space, depending on the definiteness of matrix Σ. Here as Σ is positive definite, the equation represents an ellipsoid centered at m, and its orientation in the space is determined by Σ. Specially if components in x are uncorrelated, i.e., σ_{ij} = 0 for all i ≠ j, then Σ = diag[σ_{1}^{2}, ⋯, σ_{n}^{2}] is a diagonal matrix and Eq. (B.103) becomes

(x − m)^{T} Σ^{−1}(x − m) = ∑_{i=1}^{n} \frac{(x_{i} − μ_{i})^{2}}{σ_{i}^{2}} = c_{1}  (B.104)

which represents a standard ellipsoid with all its principal semi-axes are in parallel to those of the coordinate system. Still more specially if x is standardized with m = 0 and Σ = I, then Eq. (B.103) represents a hypersphere.

Properties of Gaussian distribution

• Affine transformation of normal distributions:

If p(x) = N (m_{x}, Σ_{x}), then p(y) = p(Ax + b) = N (m_{y}, Σ_{y}), where

m_{y} = Am_{x} + b, Σ_{y} = AΣ_{x}A^{T}  (B.105)

Proof:

m_{y} = E[Ax + b] = A E[x] + b = Am_{x} + b  (B.106)

Σ_{y} = E[(y − m_{y})(y − m_{y})^{T}]
= E[[(Ax − b) − (Am_{x} − b)][(Ax − b) − (Am_{x} + b)]^{T}]
= E[(Ax − Am_{x})(Ax − Am_{x})^{T}] = A E[(x − m_{x})(x − m_{x})^{T}]A^{T}
= AΣ_{x}A^{T}  (B.107)

Some special cases:

– If A = I, then p(x + b) = N (m_{x} + b, Σ_{x})

– If b = 0, then p(Ax) = N (m_{x}, AΣ_{x}A^{T})

– If p(x) = N ([μ_{1}, ⋯, μ_{n}]^{T} , diag[σ_{1}^{2}, ⋯, σ_{n}^{2}]), then p(∑_{i=1}^{n} x_{i}) = N (∑_{i=1}^{n} μ_{i}, ∑_{i=1}^{n} σ_{i}^{2})

• Marginal distributions:

p(x_{1}, x_{2}) = p(x) = p ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]) = N ([ \begin{matrix}μ_{1} \\ μ_{2} \end{matrix} ] , [ \begin{matrix}σ_{1}^{2} & σ_{12}^{2} \\ σ_{21}^{2} & σ_{2}^{2} \end{matrix} ]) (B.108)

Applying A_{1} = [1, 0] and A_{2} = [0, 1] to x = [x_{1}, x_{2}]^{T}, we get the two marginal distributions for A_{1}x = x_{1} and A_{2}x = x_{2}, respectively:

p(x_{1}) = N (μ_{1}, σ_{1}^{2}), p(x_{1}) = N (μ_{2}, σ_{2}^{2})  (B.109)

• Product of Gaussians: Consider two Gaussians:

p_{i}(x) = N (x, m_{i}, Σ_{i}) = \frac{1}{(2π)^{n/2}|Σ_{i}|^{1/2}} exp [−\frac{1}{2}(x − m_{i})^{T} Σ_{i}^{−1}(x − m_{i})]
= \frac{1}{(2π)^{n/2}|Σ_{i}|^{1/2}} exp [−\frac{1}{2}(x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i} + m_{i}Σ_{i}^{−1}m_{i})]
= exp [−\frac{1}{2}[n log(2π) + log |Σ_{i}| + m_{i}Σ_{i}^{−1}m_{i}]] exp [−\frac{1}{2}(x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i})]
= exp [−\frac{1}{2}[x^{T} Σ_{i}^{−1}x − 2x^{T} Σ_{i}^{−1}m_{i} + c_{i}]] (i = 1, 2)  (B.110)

where

c_{i} = n log(2π) + log |Σ_{i}| + m_{i}Σ_{i}^{−1}m_{i}  (B.111)

The product of two Gaussians is

p_{1}(x)p_{2}(x) = N (x, m_{1}, Σ_{2})N (x, m_{2}, Σ_{2})
= exp [−\frac{1}{2}[x^{T} Σ_{1}^{−1}x − 2x^{T} Σ_{1}^{−1}m_{1} + c_{1} + x^{T} Σ_{2}^{−1}x − 2x^{T} Σ_{2}^{−1}m_{2} + c_{2}]]
= exp [−\frac{1}{2}[x^{T} (Σ_{1}^{−1} + Σ_{2}^{−1})x − 2x^{T} (Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) + c_{1} + c_{2}]]
= exp [−\frac{1}{2}[x^{T} Σ^{−1}x − 2x^{T} Σ^{−1}m + c + (c_{1} + c_{2} − c)]]
= N (x, m, Σ) exp(c_{1} + c_{2} − c) ∝ N (x, m, Σ)  (B.112)

Here we have defined

c = n log(2π) + log |Σ| + m^{T} Σ^{−1}m  (B.113)

Σ = (Σ_{1}^{−1} + Σ_{2}^{−1})^{−1}  (B.114)

and

Σ^{−1}m = Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}  (B.115)

i.e.,

m = Σ(Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) = (Σ_{1}^{−1} + Σ_{2}^{−1})^{−1}(Σ_{1}^{−1}m_{1} + Σ_{2}^{−1}m_{2}) (B.116)

We see that the product of Gaussians is also a scaled Gaussian.

• Marginal and conditional distributions of multivariate normal dis-

tribution

A random vector x = [x_{!}^{T} , x_{2}^{T}]^{T} with a normal distribution N (x, m, Σ) can be partitioned into two subvectors x_{1} and x_{2}:

p ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ]) = p(x) = N (x, m, Σ) = N ([ \begin{matrix}x_{1} \\ x_{2} \end{matrix} ] , [ \begin{matrix}m_{1} \\ m_{2} \end{matrix} ] , [ \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ])

(B.117)

where Σ_{ij} = Σ_{ji}^{T}, Σ_{ii} = Σ_{ii}^{T}, (i, j = 1, 2, i ≠ j). Given this normal joint distribution of x_{1} and x_{2}, we can find the normal conditional distribution of x_{i} given x_{j}:

p(x_{i}/x_{j}) = N (x_{i}, m_{i/j}, Σ_{i/j})  (B.118)

in terms of the mean vector and the covariance matrix:

\{ \begin{matrix}m_{i/j} & = & m_{i} + Σ_{ij}Σ_{jj}^{−1}(x_{j} − m_{j}) \\ Σ_{i/j} & = & Σ_{ii} − Σ_{ij}Σ_{jj}^{−1}Σ_{ji} \end{matrix}  (B.119)

Proof: The joint distribution of x can be written as:

p(x) = p(x_{1}, x_{2}) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)]
= \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}Q(x_{1}, x_{2})]  (B.120)

Here we have defined:

Q(x_{1}, x_{2}) = (x − m)^{T} Σ^{−1}(x − m)
= [(x_{1} − m_{1})^{T} , (x − m_{2})^{T}] [ \begin{matrix}Σ^{11} & Σ^{12} \\ Σ^{21} & Σ^{22} \end{matrix} ] [ \begin{matrix}x_{1} − m_{1} \\ x_{2} − m_{2} \end{matrix} ]
= (x_{1} − m_{1})^{T} Σ^{11}(x_{1} − m_{1}) + 2(x_{1} − m_{1})^{T} Σ^{12}(x_{2} − m_{2}) + (x_{2} − m_{2})^{T} Σ^{22}(x_{2} − m_{2}) (B.121)

where we have assumed

Σ^{−1} = [ \begin{matrix}Σ_{11} & Σ_{12} \\ Σ_{21} & Σ_{22} \end{matrix} ]^{−1} = [ \begin{matrix}Σ^{11} & Σ^{12} \\ Σ^{21} & Σ^{22} \end{matrix} ]  (B.122)

According to the Woodbury matrix identify (Section A.2.4), we have

Σ^{11} = (Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ_{21})^{−1} = Σ_{11}^{−1} + Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}
Σ^{22} = (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1} = Σ_{22}^{−1} + Σ_{22}^{−1}Σ_{21}(Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ_{21})^{−1}Σ_{12}Σ_{22}^{−1}
Σ^{12} = −Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1} = (Σ^{12})  (B.123)

Substituting the second expression for Σ^{11}, the first expression for Σ^{22}, and Σ^{12} into Q(x_{1}, x_{2}) we get:

Q(x_{1}, x_{2}) = (x_{1} − m_{1})^{T} [Σ_{11}^{−1} + Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}](x_{1} − m_{1})

−2(x_{1} − m_{1})^{T} [Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}](x_{2} − m_{2})

+(x_{2} − m_{2})^{T} [(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}](x_{2} − m_{2})

= (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})

+(x_{1} − m_{1})^{T} Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})

−2(x_{1} − m_{1})^{T} Σ_{11}^{−1}Σ_{12}(Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}(x_{2} − m_{2})

+(x_{2} − m_{2})^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}(x_{2} − m_{2})

= (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})

+[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})] (B.124)

The last equal sign is due to the following equations for any vectors u and v and a symmetric matrix A = A^{T}:

u^{T} Au − 2u^{T} Av + v^{T} Av = u^{T} Au − u^{T} Av − u^{T} Av + v^{T} Av
= u^{T} A(u − v) − (u − v)^{T} Av = u^{T} A(u − v) − v^{T} A(u − v)
= (u − v)^{T} A(u − v) = (v − u)^{T} A(v − u)  (B.125)

We further define

\{ \begin{matrix}b = m_{2} + Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1}) \\ A = Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12} \end{matrix}  (B.126)

and

Q_{1}(x_{1})  = (x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})
Q_{2}(x_{1}, x_{2}) = [(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]^{T} (Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12})^{−1}[(x_{2} − m_{2}) − Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1})]
= (x_{2} − b)^{T} A^{−1}(x_{2} − b)

(B.127)

then the expression above for Q(x_{1}, x_{2}) can be written as:

Q(x_{1}, x_{2}) = Q_{1}(x_{1}) + Q_{2}(x_{1}, x_{2})  (B.128)

and the joint distribution can be written as:

p(x) = p(x_{1}, x_{2}) = \frac{1}{(2π)^{n/2}|Σ|^{1/2}} exp [−\frac{1}{2}Q(x_{1}, x_{2})]
= \frac{1}{(2π)^{n/2}|Σ_{11}|^{1/2}|Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12}|^{1/2}} exp [−\frac{1}{2}Q_{1}(x_{1})] exp [−\frac{1}{2}Q_{2}(x_{1}, x_{2})]
= \frac{1}{(2π)^{p/2}|Σ_{11}|^{1/2}} exp [−\frac{1}{2}(x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})] \frac{1}{(2π)^{q/2}|A|^{1/2}} exp [−\frac{1}{2}(x_{2} − b)^{T} A^{−1}(x_{2} − b)]
= N (x_{1}, m_{1}, Σ_{11}) N (x_{2}, b, A)  (B.129)

where we have used theorem 2 in Section 7 in Appendix A:

|Σ| = |Σ_{11}| |Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12}|  (B.130)

The marginal distribution of x_{1} is

p_{1}(x_{1}) = ∫ p(x_{1}, x_{2})dx_{2} = N (x_{1}, m_{1}, Σ_{11}) ∫ N (x_{2}, b, A)dx_{2}
= N (x_{1}, m_{1}, Σ_{11}) = \frac{1}{(2π)^{p/2}|Σ_{11}|^{1/2}} exp [−\frac{1}{2}(x_{1} − m_{1})^{T} Σ_{11}^{−1}(x_{1} − m_{1})] (B.131)

and the conditional distribution of x_{2} given x_{1} is

p_{2|1}(x_{2}|x_{1}) = \frac{p(x_{1}, x_{2})}{p_{1}(x_{1})} = N (x_{2}, b, A) = \frac{1}{(2π)^{q/2}|A|^{1/2}} exp [−\frac{1}{2}(x_{2} − b)^{T} A^{−1}(x_{2} − b)]

(B.132)

with

\{ \begin{matrix}b & = & m_{2/1} = m_{2} + Σ_{21}Σ_{11}^{−1}(x_{1} − m_{1}) \\ A & = & Σ_{2/1} = Σ_{22} − Σ_{21}Σ_{11}^{−1}Σ_{12} \end{matrix}  (B.133)

Drawing samples from Gaussian distributions

We first consider drawing samples from a random variable x with a normal distribution p(x) = N (0, 1), i.e., E[x] = 0 and Var[x] = E[x^{2}] = 1. As shown in the figure, a random variable uniformly distributed in the range (0, 1) (vertical) can be mapped to another random variable x in the range (−∞, ∞) (horizontal) with a Gaussian distribution p(x) = N (0, 1), by the inverse function of its accumulative F (x) = ∫_{−∞}^{x} p(u) du. This variable x can be further converted into another variable x′ = μ + σx, whose distribution is a generic Gaussian p(x′) = N (μ, σ^{2}), with the mean and variance

E[x′] = E[μ + σx] = μ + σ E[x] = μ
Var[x′] = [(x′ − μ)^{2}] = E[σ^{2}x] = E[σ^{2}x^{2}] = σ^{2} E[x^{2}] = σ^{2} (B.134)

Using this method, we can generate a set of n normally distributed and independent random variables x_{1}, ⋯, x_{n}, each with E[x_{i}] = 0, V ar(x_{i}) = 1, and Cov[x_{i}, x_{j}] = δ_{ij}, and, based on these variables, we can further construct a random vector x = [x_{1}, ⋯, x_{n}]^{T} with a Gaussian joint distributed p(x) = N (0, I).

Given such an x with m_{x} = 0 and Σ_{x} = I, we can generate another random vector with a Gaussian distribution N(m, Σ). To do so, consider the affine transformation y = Wx + m, which also has a Gaussian distribution:

m_{y} = E[y] = Wm_{x} + m = m
Σ_{y} = WΣ_{x}W^{T} = WW^{T}  (B.135)

While m_{y} = m as desired, we still need to make sure that Σ_{y} = WW^{T} = Σ also as desired. There exist infinite matrices all satisfying the equation above. Typically, such a W can be found in either of the following two methods:

• The Cholesky decomposition: which can find an upper triangular matrix L that satisfies Σ = LL^{T} (if Σ is nonsingular).

• The eigenvalue decomposition Σ = VΛV^{T} (even if Σ is singular), where V and Λ are respectively the eigenvector and eigenvalue matrices of Σ. As Σ^{T} = Σ is symmetric and positive definite, its eigenvalues take positive real values and its eigenvectors are orthogonal to each other, i.e., V^{T} = V^{−1} is orthogonal. If we let W = VΛ^{1/2}, then we have

WW^{T} = (VΛ^{1/2}) (VΛ^{1/2})^{T} = VΛV^{T} = Σ  (B.136)

We note that although LL^{T} = WW^{T} = Σ, L found by the Cholesky decomposition method is upper triangular but W is not, L ≠ W. Also, we note that the transform based on eigenvalue decomposition is optimal in certain way, as to be discussed later.

Having found either L or W, we get y = Lx + m or y = Wx + m with p(y) = N (m, Σ) as desired.

B.1.6 Chi-Square and t-Distributions

Let {x_{1}, ⋯, x_{N}} be a set of N i.i.d. samples from a normal distribution x_{n} ∼ N (μ, σ^{2}). The mean μ and variance σ^{2} can be estimated by

μ̂_{N} = \frac{1}{N} ∑_{n=1}^{N} x_{n},  σ̂_{N}^{2} = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} − μ̂)^{2}  (B.137)

The estimated mean μ̂ and variance σ̂^{2} are in turn random variables associated with two important probability density distributions:

• Student’s t-distribution:

The standardized estimated mean in Eq. (B.137) has a standard normal distribution:

\frac{μ̂_{N} − μ}{σ/\sqrt{N}} ∼ N (0, 1)  (B.138)

When the true standard deviation σ is unknown and replaced by its estimation σ̂, then the standardized estimated mean has a Student’s t-distribution:

t = \frac{μ̂_{N} − μ}{σ̂/\sqrt{N}} ∼ T_{ν}(t) = \frac{Γ (\frac{ν+1}{2})}{\sqrt{νπ} Γ (\frac{ν}{2} )} (1 + \frac{t^{2}}{ν} )^{−(ν+1)/2}  (B.139)

where ν = N −1 is the degree of freedom and Γ(n) = (n−1)! is the gamma function, with zero mean and variance ν/(ν − 2). Specially when ν = 1, Γ((ν + 1)/2) = Γ(1) = 1, Γ(ν/2) = Γ(1/2) = \sqrt{π}, and

T_{1} = \frac{1}{π} (1 + t^{2})^{−1}  (B.140)

When ν → ∞, the t-distribution approaches the standard normal distribution:

T_{ν} ⟹^{ν→∞} N (0, 1)  (B.141)

We can further get the distribution of variable μ̂_{N} − μ (Eq. (B.71)):

μ̂_{N} − μ = \frac{σ}{\sqrt{N}} t ∼ \frac{\sqrt{N}}{σ} T_{ν}(t)  (B.142)

with zero mean and variance σ^{2}/N (Eq. (B.75)), i.e., μ̂_{N} has mean μ and variance σ^{2}/N.

When ν = N −1 → ∞, the variance of this distribution approaches zero, i.e., the estimated μ̂ approaches the true μ:

μ̂_{N} = \frac{1}{N} ∑_{n=1}^{N} x_{n} = \frac{σ}{\sqrt{N}} t − μ ⟹^{N→∞} μ  (B.143)

• Chi-square distribution: A random variable x ∼ N (μ, σ^{2}) can be standardized to become

z = \frac{x − μ}{σ} ∼ N (0, 1)  (B.144)

and the sum of squares of a set of N i.i.d. samples {z_{1}, ⋯, z_{N}} of z has a chi-square distribution:

s = ∑_{n=1}^{N} z_{n}^{2} = ∑_{n=1}^{N} (\frac{x_{n} − μ}{σ})^{2} ∼ χ^{2}(s) = \frac{s^{ν/2−1}e^{−s/2}}{2^{ν/2}Γ(ν/2)}  (B.145)

where ν = N − 1 is the degree of freedom, with mean ν and variance 2ν. When ν → ∞, the chi-square distribution tends to a normal distribution N (ν, 2ν), which has infinite variance and approaches 0.

The estimated variance σ̂^{2} in Eq. (B.137) can be written as (Eq. (B.71)):

σ̂^{2} = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} − μ̂)^{2} = \frac{σ^{2}}{N − 1} s = \frac{σ^{2}}{ν} s ∼ \frac{ν}{σ^{2}} χ^{2}  (B.146)

with the following mean and variance (Eq. (B.75)):

\frac{σ^{2}}{ν} ν = σ^{2}, (\frac{σ^{2}}{ν} )^{2} 2ν = 2σ^{4}/ν  (B.147)

When ν = N − 1 → ∞, the variance of the distribution approaches zero, i.e., the estimated σ̂^{2} approaches the true σ^{2}:

σ̂_{N}^{2} = \frac{1}{N − 1} ∑_{n=1}^{N}(x_{n} − μ̂)^{2} = \frac{σ^{2}}{ν} s ⟹^{N→∞} σ^{2}  (B.148)

B.2 Entropy and Information

B.2.1 Entropy

Entropy is a measurement of the uncertainty of the outcome of a random event, which also measures the amount of information obtained in terms of the reduction of uncertainty (due to certain communication or experiment). If the uncertainty is reduced to zero by a communication, then the amount of information obtained is exactly the same as the uncertainty before the communication.

A random event of n outcomes can be represented by a discrete random variable x with N possible values {x_{1}, ⋯, x_{n}} with the corresponding probabilities P_{i} = P (x_{i}) satisfying ∑_{i=1}^{n} P_{i} = 1. For example, n = 26 letters in the English alphabet, with different probabilities to be used (e.g., P (x =′ a′) > P (x =′ z′)).

To quantitatively measure the uncertainty of random variable x denoted by H(x), we define the uncertainty or information content (surprise) of each outcome x_{i} as

H(x_{i}) = log \frac{1}{P_{i}} = − log P_{i}, (i = 1, ⋯, n)  (B.149)

so that the desired properties listed below are satisfied:

• Uncertainty is inversely related to the probability, i.e., a more probable outcome is less uncertain than a less probable outcome. If P_{i} > P_{j}, then

H(x_{i}) = − log P_{i} < H(x_{j}) = log P_{j}  (B.150)

• Specially, an certain outcome x with P (x_{i}) = 1 has zero uncertainty, and an impossible outcome x with P (x_{j}) = 0 has infinite uncertainty:

H(x_{i}) = − log P_{i} = − log 1 = 0, H(x_{j}) = − log P_{j} = − log 0 = ∞

(B.151)

• The total uncertainty of x is the expectation of the uncertainties of the n outcomes:

H(x) = E[H(x_{i})] = E [− log P_{i}] = − ∑_{i=1}^{n} P_{i} log P_{i}  (B.152)

The total uncertainty H(x) defined above, called the entropy, measures the total uncertainty of the random event represented by x, which is also the total amount of information contained in x. Once the outcome of the random event is known, the information contained in x is gained, and the uncertainty is reduced to zero.

The n outcomes of x can be binary encoded so that each outcome x_{i} is represented by a sequence of 0’s and 1’s. To minimizes the total number of bits needed to encode x, we encode x_{i} with P_{i} by H(x_{i}) = − log_{2} P_{i} bits, so that a more probable outcome x_{i} with greater P_{i} (less uncertain and containing less information) is encoded by fewer bits, and a less probable outcome x_{j} with smaller P_{j} (more uncertain and containing more information) is encoded by more bits. Then the expected number of bits needed to encode x is minimized.

We can show that a uniform distribution of n = 2^{m} equally likely outcomes with p_{i} = 1/n (i = 1, ⋯, n) has the maximum entropy. This is a constrained maximization problem which can be solved by Lagrange multiplier method:

\frac{∂}{∂P_{j}} [− ∑_{i=1}^{n} P_{i}log P_{i} + λ (1 − ∑_{i=0}^{n} P_{i})] = 0

= −\frac{∂}{∂P_{j}} P_{j} log P_{j} − λ = −(log P_{j} + 1) − λ = 0  (B.153)

Solving this we get

P_{j} = 2^{−λ−1} (j = 1, ⋯, N)  (B.154)

which must satisfy

∑_{j=1}^{n} P_{j} = ∑_{j=1}^{n} 2^{−λ−1} = 2^{−λ−1}n = 1  (B.155)

Solving for λ we get

λ = log n − 1  (B.156)

Substituting back into P_{j} we get

P_{j} = 2^{−λ−1} = 2^{− log n} = \frac{1}{n}, (j = 1, ⋯, n)  (B.157)

The corresponding maximum entropy can be obtained as

H_{max}(E_{1}, ⋯, E_{n}) = − ∑_{i=1}^{n} \frac{1}{n}log_{2} \frac{1}{n} = log_{2}n = m  (B.158)

In this case, each outcome x_{i} is encoded by

log_{2} \frac{1}{P_{i}} = log_{2} (\frac{1}{1/2^{n}} ) = log_{2} 2^{n} = n  (B.159)

bits, and the total number of bits needed to encode this random event is

− ∑_{i=1}^{n} P_{i} log_{2} P = − ∑_{i=1}^{n} \frac{1}{n} log_{2} \frac{1}{2^{m}} = ∑_{i=1}^{n} \frac{1}{n} log_{2} 2^{m} = m  (B.160)

If n = 2^{m} = 2^{3} = 8, P_{i} = 1/2^{3} = 1/8, n = 3 bits are needed to encode the n = 8 possible outcomes.

For example, in the special case of n = 2 outcomes:

• If P_{1} = 1 and P_{2} = 0, we have the minimum uncertainty zero (x log_{2} x|_{x=0} = 0), i.e., 0 bit is needed to encode a sure event:

H = −1 log_{2} 1 − 0 log_{2} 0 = 0  (B.161)

• If P_{1} = 0.1, P_{2} = 0.9, the uncertainty is:

H = −(0.1 log_{2} 0.1 + 0.9 log_{2} 0.9) = 0.47  (B.162)

• If P_{1} = 0.2, P_{2} = 0.8, the uncertainty is:

H = −(0.2 log_{2} 0.2 + 0.8 log_{2} 0.8) = 0.72  (B.163)

• If P_{1} = 0.3, P_{2} = 0.7, the uncertainty is:

H = −(0.3 log_{2} 0.3 + 0.7 log_{2} 0.7) = 0.88  (B.164)

• If P_{1} = 0.4, P_{2} = 0.6, the uncertainty is:

H = −(0.4 log_{2} 0.4 + 0.6 log_{2} 0.6) = 0.97  (B.165)

• If P_{1} = P_{2} = 0.5 = 1/2, we have the maximum uncertainty H = 1, i.e., 1 bit (0 or 1) is needed to encode the event:

H = −\frac{1}{2} log_{2} \frac{1}{2} − \frac{1}{2} log_{2} \frac{1}{2} = \frac{1}{2} + \frac{1}{2} = 1  (B.166)

The discussion above for a discrete random variable x can be extended to a continuous random variable x with pdf p(x), of which the differential entropy is defined as

H(p) = − ∫ p(x) log p(x) dx  (B.167)

The perplexity of a discrete probability distribution P is simply defined as 2^{H(p)}.

We consider the pdf p(x) with maximum entropy under each of the following two conditions:

• The distribution p(x) is non-zero only inside a closed region x ∈ [a, b].

To find the pdf p(x) that maximizes H(p) under the condition ∫_{a}^{b} p(x) dx = 1, we construct its Lagrangian function

L(p) = H(p) + λ (1 − ∫_{a}^{b} p(x) dx) = ∫_{a}^{b} [−p(x) log p(x) − λp(x)] dx + λ

(B.168)

and set its derivative with respect to p(x) (with a specific value of x) to zero:

\frac{d L(p)}{d p(x)} = −(log p(x) + 1) − λ = 0  (B.169)

Solving for p(x) we get

p(x) = e^{−λ−1}  (B.170)

which must satisfy

∫_{−∞}^{∞} p(x) dx = ∫_{a}^{b} e^{−λ−1} dx = e^{−λ−1}(b − a) = 1  (B.171)

i.e.,

p(x) = e^{−λ−1} = \frac{1}{b − a}  (B.172)

This is a uniform distribution over [a, b]. The corresponding maximum entropy is

H(p) = − ∫_{a}^{b} \frac{1}{b − a} log (\frac{1}{b − a}) = log(b − a)  (B.173)

• The distribution p(x) is non-zero for all x ∈ (−∞, ∞) with a known variance

σ^{2} = ∫_{−∞}^{∞}(x − μ)^{2}p(x) dx  (B.174)

To find the pdf p(x) with σ^{2} that maximizes H(p), we consider the following Lagrangian function

L(p) = − ∫_{−∞}^{∞} p(x) log p(x) dx+λ_{1} (1 − ∫_{−∞}^{∞} p(x) dx)+λ_{2} (σ^{2} − ∫_{−∞}^{∞}(x − μ)^{2} p(x) dx)

(B.175)

and set its derivative with respect to p(x) (evaluated at certain x) to zero:

\frac{d L(p)}{d p(x)} = −(log p(x) + 1) − λ_{1} − λ_{2}(x − μ)^{2} = 0  (B.176)

Solving for p(x) we get

p(x) = e^{−(λ_{1}+λ_{2}(x−μ)^{2}+1)} = e^{−(λ_{1}+1)}e^{−λ_{2}(x−μ)^{2}}  (B.177)

which must satisfy the two constraints:

1 = ∫_{−∞}^{∞} p(x) dx = e^{−λ_{1}−1} ∫_{−∞}^{∞} e^{−λ_{2}(x−μ)^{2}}dx = e^{−(λ_{1}+1)}\sqrt{\frac{π}{λ_{2}}} (B.178)

and

σ^{2} = ∫_{−∞}^{∞}(x − μ)^{2}p(x) dx = e^{−λ_{1}−1} ∫_{−∞}^{∞} e^{−λ_{2}(x−μ)^{2}}(x − μ)^{2} dx
= e^{−λ_{1}−1} \frac{1}{2}\sqrt{\frac{π}{λ_{2}^{3}}} = e^{−λ_{1}−1} \frac{1}{2λ_{2}} \sqrt{\frac{π}{λ_{2}}}  (B.179)

Comparing the two equations above, we get

λ_{2} = \frac{1}{2σ^{2}}  (B.180)

Substituting this into Eq. (B.178), we get

e^{−(λ_{1}+1)}\sqrt{2πσ^{2}} = 1,  i.e.  e^{−(λ_{1}+1)} = \frac{1}{\sqrt{2πσ^{2}}}  (B.181)

Substituting this together with λ_{2} = 1/2σ^{2} into p(x) we get

p(x) = e^{−(λ_{1}+1)}e^{−λ_{2}(x−μ)^{2}} = \frac{1}{\sqrt{2πσ^{2}}} e^{− \frac{(x−μ)^{2}}{2σ^{2}}} = N (x, μ, σ^{2}) (B.182)

This is the normal or Gaussian distribution and the corresponding maximum entropy is

H(g) = − ∫_{−∞}^{∞} g(x) log g(x) dx = − ∫_{−∞}^{∞} g(x) log (\frac{1}{\sqrt{2πσ^{2}}} e^{−(x−μ)^{2}/2σ^{2}}) dx
= − ∫_{−∞}^{∞} g(x) (log \frac{1}{\sqrt{2πσ^{2}}} − \frac{(x − μ)^{2}}{2σ^{2}} ) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∫_{−∞}^{∞}(x − μ)^{2}g(x) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} σ^{2} = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} log(e)
= \frac{1}{2} log(2πeσ^{2}) = \frac{1}{2} log(2πe) + log σ = 1.4189 + log σ  (B.183)

In particular, the entropy of a Gaussian distribution with unit variance σ = 1 is H(g) = 1.4189.

The uncertainty of a distribution p(x) can be measured by both its variance σ^{2} = ∫ (x − μ)^{2}p(x) dx and its entropy H = − ∫ p(x) log p(x) dx. However, they are also different in that σ^{2} represents the spread of p(x) while H captures the shape of p(x). In multimodal distribution (with two or more modes), H is a better measurement as it increases when there exist multiple modes while σ^{2} is not sensitive to them.

In data analysis and modeling, if no additional knowledge is available regarding the observed data other than the variance (or covariance for multivariate data), the normal distribution with maximum entropy should be used as leastinformative default to model the data, as it will impose the least amount of unsupported constraint and thereby causing minimum bias. This is the principle of maximum entropy.

• Joint Entropy The joint entropy of two discrete random variables x and y is defined as:

H(x, y) = − E[log P (x_{i}, y_{j})] = − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i}, y_{j}) (B.184)

More generally, the joint entropy of n discrete random variables in x = [x_{1}, ⋯, x_{n}]^{T} is defined as:

H(x) = H(x_{1}, ⋯, x_{n}) = − E[log P (x_{1}, ⋯, x_{n})]
= − ∑_{x_{1}} ⋯ ∑_{x_{n}} P (x_{1}, ⋯, x_{n}) log P (x_{1}, ⋯, x_{n})
= − ∑_{x_{1}} ⋯ ∑_{x_{n}} P (x) log P (x)  (B.185)

The differential entropy is

H(x) = H(x_{1}, ⋯, x_{n}) = − ∫ ⋯ ∫ p(x_{1}, ⋯, x_{n}) log P (x_{1}, ⋯, x_{n})dx_{1} ⋯ dx_{n}
= − ∫ p(x) log P (x)dx  (B.186)

• Entropy of function of random variables

Given the entropy H(x) of x, we can further find the entropy of a function y = f(x) of x. We first recall that the pdf of y can be expressed in terms of that of x in Eq. (B.83): p_{Y} (y) = | det J_{f^{−1}}|p_{X}(x). We can then find H(y) directly as:

H(y) = − ∫ p_{y}(y) log p_{y}(y)dy
= − ∫ | det J_{f^{−1}}| p_{X}(x) log (| det J_{f^{−1}}| p_{X}(x)) | det J_{f}| dx
= − ∫ p_{X}(x) [log | det J_{f^{−1}}| + log p_{X}(x)] dx
= − ∫ p_{X}(x) log p_{X}(x) dx − ∫ p_{X}(x) log | det J_{f^{−1}}| dx
= H(x) + E [log | det J_{f}|]  (B.187)

Here we have used integration by substitution and also the inverse function theorem stating that given the Jacobian J_{f} of a function y = f(x), the Jacobian of the inverse function x = f^{−1}(y) is J_{f^{−1}} = J_{f}^{−1}. Also as in general det(A^{−1}) = (det(A))^{−1}, we have | det J_{f^{−1}}| | det J_{f}| = 1.

Specially for a linear function y = f(x) = Ax, we have J_{f} = df(x)/dx = A and J_{f^{−1}} = A^{−1}, then we have

H(y) = H(x) + log | det A|  (B.188)

• Conditional Entropy

The conditional entropy H(y|x) measures the uncertainty of a random variable y given the value of another random variable x, or the amount of information gained once the outcome of y is known, given the outcome of x.

Let H(y|x = x_{i}) be the entropy of y conditioned on x = x_{i}:

H(y|x = x_{i}) = − ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i})  (B.189)

Then the conditional entropy H(y|x) is defined as the average of H(y|x = x_{i}) over all outcomes of x:

H(y|x) = E_{x}[H(y|x = x_{i})] = ∑_{i} P (x_{i})H(y|x = x_{i})
= − ∑_{i} P (x_{i}) ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i}) = − ∑_{i} ∑_{j} P (x_{i})P (y_{j}|x_{i}) log P (y_{j}|x_{i})
= − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j}|x_{i})  (B.190)

The last equality is due to P (x, y) = P (y|x)P (x).

Theorem:

H(y|x) = − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j}|x_{i}) = ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i})}{P (x_{i}, y_{j})}
= − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i}, y_{j}) + ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (x_{i})
= H(x, y) + ∑_{i} P (x_{i}) log P (x_{i}) = H(x, y) − H(x)  (B.191)

Similarly, we also have H(x|y) = H(x, y) − H(y). We therefore have

H(x, y) = H(y|x) + H(x) = H(x|y) + H(y)  (B.192)

Substituting this into the previous equation we get Bayes’s rule for conditional entropy:

H(y|x) = H(x|y) + H(y) − H(x)  (B.193)

B.2.2 Kullback-Leibler Divergence and Mutual Information

• Cross Entropy

When the true distribution P is unknown, the encoding of x can be based on another distribution Q as a model that approximates P. Then the average of the total number of bits needed is called the cross-entropy:

H(P, Q) = ∑_{i} P_{i} log_{2} \frac{1}{Q_{i}} = − ∑_{i} P_{i} log Q_{i} = −E_{P} [log Q]  (B.194)

which can be considered as the expectation of the logarithmic model probability log Q_{i} with respect to the unknown ground truth probability P_{i}.

Gibbs’ inequality: The cross entropy H(P, Q) is no smaller than the entropy H(P):

H(P) = − ∑_{i} P_{i} log P_{i} ≤ H(P, Q) = − ∑_{i} P_{i} log Q_{i}  (B.195)

The equality holds H(P, Q) = H(P) if and only if Q = P. For the model Q to be as close to the ground truth P as possible, we need to minimize the cross entropy H(P, Q).

Proof: Based on inequality log x ≤ x − 1, we have

∑_{i} P_{i} log (\frac{Q_{i}}{P_{i}} ) ≤ ∑_{i} P_{i} (\frac{Q_{i}}{P_{i}} − 1) = ∑_{i} Q_{i} − ∑_{i} P_{i} = 0 (B.196)

i.e.,

∑_{i} P_{i} log (\frac{Q_{i}}{P_{i}} ) = ∑_{i} P_{i} log (\frac{1}{P_{i}} ) + ∑_{i} P_{i} log Q_{i} = H(P) − H(P, Q) ≤ 0

(B.197)

We see that the encoding of x based on model distribution Q always requires more bits than that based on the true distribution P.

• Kullback-Leibler Divergence (Relative Entropy)

The Kullback-Leibler (KL) divergence or relative entropy is the difference between the cross entropy H(P, Q) and the entropy H(P):

D_{KL}(P ||Q) = H(P, Q) − H(P) = −E_{P} [log Q] − E_{P} [log P]
= − ∑_{i} P_{i} log Q_{i} − (− ∑_{i} P_{i} log P_{i})
= ∑_{i} P_{i}(log P_{i} − log Q_{i}) = ∑_{i} P_{i} log (\frac{P_{i}}{Q_{i}} ) ≥ 0(B.198)

The KL-divergence represents the number of extra bits needed to encode x based on Q instead of P, or a measure of the error of using Q to approximate P, in terms of the amount of information lost, due to the inaccuracy of the model. In order to obtain the best model Q that optimally approximates P, we need to minimize D_{KL}(P ||Q).

In general, D_{KL}(P ||Q) ≠ D_{KL}(Q||P), the KL-divergence is not a distance metric as it is not symmetric (hence the name divergence rather than distance). The KL-divergence can be symmetrized to become the Jensen–Shannon (JS) divergence:

JS(P ||Q) = \frac{1}{2} [D_{KL}(P ||M) + D_{KL}(Q||M)] , where M = \frac{P + Q}{2}

(B.199)

As JS divergence is symmetric, it can be used as a distance measure for the similarity between two distributions P and Q.

All discussions above for discrete variables can be generalized to continuous variables with summations replaced by integrals.

In particular, consider the cross-entropy of a Gaussian g(x) and an arbitrary pdf p(x) both with the same variance σ^{2}:

H(p, g) = − ∫ p(x) log g(x) dx = − ∫ p(x) (log \frac{1}{\sqrt{2πσ^{2}}} − \frac{(x − μ)^{2}}{2σ^{2}} ) dx
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} ∫ (x − μ)^{2}p(x) dx = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2σ^{2}} σ^{2}
= \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} = \frac{1}{2} log(2πσ^{2}) + \frac{1}{2} log e = \frac{1}{2} log(2πeσ^{2}) = H(g) (B.200)

We see that the cross-entropy H(p, g) of g(x) and p(x) is the same as the entropy of g(x). But as we also know H(p, g) ≥ H(p), we have H(g) = H(p, g) ≥ H(p). We therefore conclude that among all pdfs p(x) with the same variance σ^{2}, the Gaussian N(x, μ, σ^{2}) has the maximum entropy.

• Mutual Information

Mutual information I(x) of all components of a random vector x = [x_{1}, ⋯, x_{n}]^{T} measures the information shared by its components. It is defined as the KL-divergence between P (x) = P (x_{1}, ⋯, x_{n}) and ∏_{i=1}^{n} P (x_{j}):

I(x) = I(x_{1}, ⋯, x_{n}) = D_{KL} (P (x)|| ∏_{j=1}^{n} P (x_{j}))

= ∑_{i_{1}} ⋯ ∑_{i_{n}} P (x) log (\frac{P (x)}{∏_{j=1}^{n} P (x_{j})})

= ∑_{i_{1}} ⋯ ∑_{i_{n}} P (x) log P (x) − ∑_{i_{1}} ⋯ ∑_{i_{n}} P (x) ∑_{j=1}^{n} log P (x_{j})

= −H(x) − ∑_{j=1}^{n} [∑_{i_{1}} ⋯ ∑_{i_{n}} P (x) log P (x_{j})]

= ∑_{j=1}^{n} H(x_{j}) − H(x)  (B.201)

i.e., the mutual information is the error of using P (x) P (y) to model the joint probability P (x, y). When all n variables x_{1}, ⋯, x_{n} are independent, i.e., P (x_{1}, ⋯, x_{n}) = ∏_{j=1}^{n} P (x_{j}), then their mutual information is zero.

Specially the mutual information between two variables X and Y is

I(x, y) = ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i}, y_{j})}{P (x_{i})P (y_{j})}
= ∑_{i} ∑_{j} P (x_{i}, y_{j}) log \frac{P (x_{i}, y_{j})}{P (x_{i})} − ∑_{i} ∑_{j} P (x_{i}, y_{j}) log P (y_{j})

= ∑_{i} ∑_{j} P (y_{j}|x_{i})P (x_{i}) log P (y_{j}|x_{i}) − ∑_{j} log P (y_{j}) (∑_{i} P (x_{i}, y_{j}))
= ∑_{i} P (x_{i}) ∑_{j} P (y_{j}|x_{i}) log P (y_{j}|x_{i}) − ∑_{j} P (y_{j}) log P (y_{j})
= −H(y|x) + H(y) = H(y) − H(y|x)
= H(x, y) − H(y|x) − H(x|y) = H(x) + H(y) − H(x, y)  (B.202)

The last two qualities are due to Eq. (B.192) above. These relationships can be summarized in the following diagram:

B.2.3 Entropy Maximization

There may be the need to find an unknown distribution p(x) of a random variable x based on the averages of a set of m observed independent experiment results f_{i}(x):

E[f_{i}(x)] = ∫_{−∞}^{∞} p(x)f_{i}(x) dx = c_{i} (i = 1, ⋯, m)  (B.203)

Among all (infinite number of) possible density functions all satisfying these equations as well as ∫_{−∞}^{∞} p(x) dx = 1, the one with the maximum entropy H(p) is of the most interest as it allows maximum uncertainty and therefore imposes least amount of additional constraints. We can show that such a distribution has to take the following form:

p(x) = exp (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) = A exp (∑_{i=1}^{m} λ_{i}f_{i}(x))  (B.204)

where λ_{0}, ⋯, λ_{m} and A = e^{λ_{0}} are constants with values for p(x) to satisfy Eq. (B.203).

To find this distribution p(x) with maximal H(p) and satisfying the constraints imposed by Eq. (B.203) as as well as ∫_{−∞}^{∞} p(x) dx = 1 required of all probability density distributions, we need to solve the constrained optimization problem with the Lagrangian:

L(p) = ∫_{−∞}^{∞} p(x) log p(x) dx+λ_{0} (∫_{−∞}^{∞} p(x)dx − 1)+∑_{i=1}^{m} λ_{i} (∫_{−∞}^{∞} p(x)f_{i}(x) dx − 1)

(B.205)

We then take its derivative with respect to p(x) corresponding to a specific value of x (i.e., x is treated as constant and the integral is ignored), and set it to zero:

\frac{d}{dp} L(p) = \frac{d}{dp} (p log p) + λ_{0} \frac{d}{dp} (p − 1) + ∑_{i=1}^{m} λ_{i} \frac{d}{dp} (pf_{i}(x) − 1)

= log p(x) + 1 − λ_{0} − ∑_{i=1}^{m} λ_{i}f_{i}(x) = 0  (B.206)

Solving this equation we get the optimal distribution in the form of Eq. (B.204):

p(x) = exp (λ_{0} − 1 + ∑_{i=1}^{m} λ_{i}f_{i}(x)) = A exp (∑_{i=1}^{m} λ_{i}f_{i}(x))  (B.207)

where A = exp(λ_{0} − 1).

Alternatively, the same result can also be obtained by comparing the entropies of an arbitrary density function q(x), and another distribution p(x) in the form of Eq. (B.204), both assumed to satisfy Eq. (B.203), i.e.,

∫ p(x)f_{i}(x) dx = ∫ q(x)f_{i}(x) dx = c_{i} (i = 1, ⋯, m)  (B.208)

We now find H(q(x)) and compare it with H(p(x)):

H(q) = − ∫ q(x) log q(x) dx = − ∫ q(x) log (\frac{q(x)}{p(x)}p(x)) dx
= −D_{KL}(q||p) − ∫ q(x) log p(s) dx ≤ − ∫ q(x) log p(s) dx
= − ∫ q(x) (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) dx = − ∫ p(x) (λ_{0} + ∑_{i=1}^{m} λ_{i}f_{i}(x)) dx
= − ∫ p(x) log p(x) dx = H(p)  (B.209)

where we have used the fact that D_{KL}(q||p) ≥ 0 and the result in Eq. (B.208). This result again indicates that the distribution with maximal H(p) and satisfying Eq. (B.203) has to take the form in Eq. (B.204). The specific values of the m + 1 constants λ_{0}, ⋯, λ_{m} can be found by solving the m + 1 equations including Eq. (B.203) and ∫_{−∞}^{∞} p(x) dx = 1.

If the distribution p(x) in question is further required to have zero mean and unit variance, i.e.,

∫ p(x)x dx = 0,  ∫ p(x)x^{2} dx = 1  (B.210)

we can include two additional terms in Eq. (B.203) with f_{m+1}(x) = x, f_{m+2} = x^{2} with c_{m+1} = 0, c_{m+2} = 1 so that

∫ p(x)f_{m+1}dx = ∫ p(x)x dx = 0, ∫ p(x)f_{m+2}dx = ∫ p(x)x^{2} dx = 1

(B.211)

B.2.4 Kurtosis

The kurtosis of any random variable x with mean μ_{x} = E[x] and variance σ_{x}^{2} = Var(x) = E[(x − μ)^{2}] is defined as the fourth standardized moment:

Kurt(x) = E[z^{4}] = E [(\frac{x − μ_{x}}{σ_{x}} )^{4}] = \frac{E[(x − μ_{x})^{4}]}{(E[x − μ_{x}])^{4}}  (B.212)

where z = (x − μ_{x})/σ_{x} is the standardized version of x with zero-mean μ_{z} = E[z] = 0 and unity variance σ_{z}^{2} = Var(z) = E[(z − μ_{z})^{2}] = 1. As the expectation of z^{4}, the kurtosis is relatively insensitive to values close to its zero mean inside the range (−1, 1), but more sensitive to the outliers far away from its zero mean, i.e., kurtosis represents the ”tailedness” of the density distribution of the variable x.

The excess kurtosis defined as Kurt(x) − 3 is more widely used for comparison among different density distributions in terms of their tailedness. A given distribution is

• Leptokurtic (super-Gaussian) if Kurt(x) − 3 > 0 (with heavier tails)

• Mesokurtic if Kurt(x) − 3 = 0

• Platykurtic (sub-Gaussian) if Kurt(x) − 3 < 0 (with lighter tails)

The figure below is a comparison of a set of distributions:

• Laplace distribution:

p(x) = \frac{1}{\sqrt{2}σ} exp (−\sqrt{2} (\frac{|x − μ|}{σ}))  (B.213)

• Hyperbolic secant distribution:

p(x) = \frac{1}{2σ} sech (\frac{π}{2} (\frac{x − μ}{σ}))  (B.214)

• Logistic distribution:

p(x) = \frac{e^{−(x−μ)/s}}{s(1 + e^{−(x−μ)/s})^{2}} = \frac{1}{4s} sech^{2} (\frac{x − μ}{2s})  (B.215)

• Normal distribution:

p(x) = \frac{1}{\sqrt{2πσ^{2}}} exp (−\frac{1}{2} (\frac{x − μ}{σ})^{2})  (B.216)

• Wigner semicircle distribution:

p(x) = \{ \begin{matrix}\sqrt[\frac{2}{πR^{2}}]{R^{2} − x^{2}} & −R ≤ x ≤ R \\ 0 & otherwise \end{matrix}  (B.217)

• uniform distribution:

p(x) = \{ \begin{matrix}1/(b − a) & a ≤ x ≤ b \\ 0 & otherwise \end{matrix}  (B.218)

Laplace Hyperbolic Secant

Logistic^{∗}

Normal

Semicircle^{∗∗}

Uniform

Mean  μ  μ  μ  μ  0  (a + b)/2
Variance  σ^{2}  σ^{2}  σ^{2}  σ^{2}  σ^{2}  (b − a)^{2}/12
Entropy  ln(\sqrt{2}σe)  1.166  ln(\sqrt{3}σ/π) ln(2πeσ^{2}) ln(2πσ) − 1/2 ln(b − a)
E-Kurtosis  3  2  1.2  0  −1  −1.2

(B.219)

* σ = sπ/\sqrt{3}, ** σ = R/2.

B.3 Bayesian Inference

B.3.1 Frequentist versus Bayesian Statistics

In machine learning it is commonly the case that all observed data D are interpreted probabilistically, i.e., all samples in dataset X are assumed to be random vectors with certain probability distributions p(x). While building a hypothetical model parameterized by θ to fit the data, we can used either of two different approaches, the frequentist approach that treats all samples in data D as random variables and assigns probabilities to them, and the Bayesian approach that also treats the model parameters θ as random variables and assigns probabilities to them as well as to data D. More specifically, the Bayesian approach is based on a prior probability p(θ) of the parameter before observing any data, which is then improved to become the posterior probability p(θ|D) after observing the data D.

Consider, as a simple example, the estimation of the ratio θ of red balls in a box containing a large number of balls based on the observed data D: taking n random samples from the box (one at a time with replacement) and finding k ≤ n of them red.

• The frequentist approach:

The probability of finding k red balls in n samples is P (D) = P (n, k) = θ^{k}(1 − θ)^{n−k} (Bernoulli distribution), which, as the probability of an event that has already happened, should be the maximum among all possible values of θ, i.e., it should satisfy d P (D)/dθ = 0 or equivalently

\frac{d}{dθ} ln P (D) = \frac{d}{dθ} [k ln θ + (n − k) ln(1 − θ)] = \frac{k}{θ} − \frac{n − k}{1 − θ} = 0 (B.220)

Solving this equation we get the frequentist model parameter θ = k/n.

• The Bayesian approach:

The prior probability of θ has a uniform distribution P (θ) = 1, as all values θ ∈ [0, 1] are equally likely without knowledge from observing any data (P (θ) = 0 for any θ < 0 or θ > 1). The likelihood of θ is the probability of observing data D (in terms of n and k) conditioned on a given θ, i.e., p(D|θ) = p(n, k|θ) = θ^{k}(1 − θ)^{n−k}. Now we can further find the posterior:

p(θ|D) = \frac{p(D|θ) p(θ)}{p(D)} = \frac{p(D|θ) p(θ)}{∫_{0}^{1} p(D|θ)p(θ)dθ}  (B.221)

Substituting into this expression p(θ), p(D|θ) and the denominator (a beta function)

∫_{0}^{1} p(D|θ)p(θ)dθ = ∫_{0}^{1} θ^{k}(1 − θ)^{n−k}dθ = \frac{k!(n − k)!}{(n + 1)!}  (B.222)

we get

p(θ|D) = \frac{(n + 1)!}{k!(n − k)!}θ^{k}(1 − θ)^{n−k}  (B.223)

based on which we further find the expectation:

E[θ] = ∫_{0}^{1} θ p(θ|D)dθ = \frac{(n + 1)!}{k!(n − k)!} ∫_{0}^{1} θ^{k+1}(1 − θ)^{n−k}
= \frac{(n + 1)!}{k!(n − k)!} \frac{(k + 1)!(n − k)!}{(n + 2)!} = \frac{k + 1}{n + 2}  (B.224)

This is called Laplace’s rule of succession.

For example, when n = 2 there exist three possible outcomes:

k Frequentist θ

Bayesian θ

0  0  1/4
1  1/2  1/2
2  1  3/4

(B.225)

It is obvious that for such a small sample set, the frequentist results are more absolute (k = 0 and k = 2) and less reliable, while the Bayesian results are less absolute and therefore more reliable. But these two types of models tend to be the same when n approaches infinity.

As an example with a very large n, consider the sunrise problem for finding the probability that the sun will rise tomorrow, given that it has risen every day for the past 5,000 years, i.e., k = n = 5000 × 365.25. This probability is θ = k/n = 1 according to the frequentist model, but θ = (k + 1)/(n + 2) = 0.99999945 according to Eq. (B.224) by Laplace.

B.3.2 Bayesian Inference

The method of Bayesian inference based on Bayes’ theorem given in Eq. (B.12) is widely used in many machine learning algorithms. In general, Bayesian inference deduces a hypothesis H, a model of some observed dataset D in terms of its probability distribution. Bayesian inference can be an iterative process that updates the model parameters as more data becomes available. Bayesian inference can be formulated in words as

posterior = \frac{likelihood × prior}{evidence}  (B.226)

or symbolically

p(H|D) = \frac{p(D|H) p(H)}{p(D)} ∝ p(D|H) p(H)  (B.227)

where p(H) is the prior probability of the hypothesis H based on some prior knowledge without observation of any data, p(H|D) is the posterior probability of H based on the observation of some data D, p(D|H) is the likelihood of H, proportional to the probability of observing D given H, and p(D) is the probability of observing data D, which can be considered as a normalizing factor and is independent of H in question, and can therefore be dropped. Specifically, data D and hypothesis H take the following forms:

• Data D: a collection of data points in a d-dimensional feature space, represented in a data matrix X = [x_{1}, ⋯, x_{N}] containing a set of N independent samples of a vector x = [x_{1}, ⋯, x_{d}]^{T}, randomly and uniformly drown from a population with certain probability density distribution. Such samples are therefore independent and identically distributed (i.i.d.). In supervised learning, the given dataset D, now called training set, also include a vector y = [y_{1}, ⋯, y_{N}]^{T}, of which component y_{i} is a label for the corresponding x_{i} in X.

• Hypothesis H: a specific probabilistic model p(x|θ) for the dataset, parameterized by θ_{1}, ⋯, θ_{M} represented as a vector θ = [θ_{1}, ⋯, θ_{M}]^{T}, to be estimated based on the observed dataset X.

Bayes inference can now be formulated as a process for estimating the model parameter θ based on the given dataset X (possibly also y), and Eq. (B.227) can be written as:

p(θ|X) = \frac{p(X|θ) p(θ)}{p(X)} = \frac{p(X|θ) p(θ)}{∑_{θ} p(X|θ) p(θ)} ∝ p(X|θ) p(θ)  (B.228)

where

• p(θ): The a priori probability, or the simply prior, of parameter θ based on any prior knowledge before observing data X.

• L(θ|X) ∝ p(X|θ): The likelihood of the model parameter θ (hypothesis), given the observed data X, measured in terms of the probability of observing X conditioned on a value of θ. In maximum likelihood estimation, we need to find the optimal θ that maximizes the likelihood, L(θ|X) does not need to be normalized as p(X|θ) If the data samples in X = [x_{1}, ⋯, x_{N}] are independently drawn from the same pdf, i.e., they are i.i.d. samples, then

p(X|θ) = ∏_{n=1}^{N} p(x_{n}|θ)  (B.229)

• p(X) = ∑_{θ} p(X|θ)p(θ): The distribution of data D while the parameter θ is marginalized. It can also be considered as a normalization factor to guarantee that the resulting posterior is a probability satisfying ∑_{θ} p(θ|X) = 1. As p(X) is a constant independent of θ in question, it is can be dropped for convenience when maximizing the posterior.

• p(θ|X): The a posteriori probability, or simply the posterior, of the parameter θ, after observing data X.

There are two methods typically used for the estimation of the parameters in θ based on the observed data X:

• Maximum Likelihood Estimation (MLE) seeks to find θ that maximizes the

likelihood:

θ_{MLE} = arg max_{θ} p(X|θ)  (B.230)

• Maximum A Posteriori (MAP) seeks to find θ that maximizes the posterior:

θ_{MAP} = arg max_{θ} p(θ|X) ∝ p(X|θ) p(θ)  (B.231)

The difference between the two methods is whether certain prior knowledge is available regarding the distribution p(θ) of the parameters before observing any data. If no such prior knowledge is available, then we can only assume all possible values of the parameters are equally likely, i.e., the prior probability p(θ) = constant is a uniform distribution, and the two methods will produce the same result. On the other hand, if we have some prior knowledge that the parameters obey certain non-uniform distribution p(θ), e.g., they are normally distributed with certain mean and covariance, then the MAP method may produce more accurate estimation.

When we maximize either the likelihood or the posterior as a function of θ, we can equivalently maximize its logarithm (a monotonic function) for mathematical convenience, so that the two methods above can be expressed as

• MLE:

θ_{MLE} = arg max_{θ} log p(X|θ) = arg max_{θ} log (∏_{n=1}^{N} p(x_{n}|θ))

= arg max_{θ} (∑_{n=1}^{N} log p(x_{n}|θ))  (B.232)

• MAP:

θ_{MAP} = arg max_{θ} log (p(X|θ) p(θ))
= arg max_{θ} (∑_{n=1}^{N} log p(x_{n}|θ) + log p(θ))  (B.233)

Both MLE and MAP seek to find some specific values for the parameters in θ that maximize certain function of the parameters, such as the likelihood or the posterior of θ, or their logarithm functions. In certain situations, we can instead find the parameter θ that maximizes the expectation of the function to be maximized, denoted by Q(θ), by the method called expectation-maximization (EM), in an iteration of the following two steps:

• The E-step: find the expectation Q(θ) based on the previous parameter values θ_{n};

• The M-step: find the updated parameter values θ_{n+1} that maximize the expectation Q(θ).

B.3.3 Maximum Likelihood Estimation of Model Parameters

Given an observed dataset X = [x_{1}, ⋯, x_{N}] containing N independent and identically distributed (i.i.d.) sample data points, we can build a statistical model p(x|θ), of which the parameter θ containing a set of model parameters is to be estimated based on the observed dataset X by method such as maximum likelihood estimation (MLE).

Specifically, we let θ′ be the current estimation of the true model parameter θ, and find its likelihood and log-likelihood:

L(θ′|x) ∝ p(x|θ′),  l(θ′|x) = log p(x|θ′)  (B.234)

We further consider the expectation of the log-likelihood with respect to the true distribution p(x|θ) is

E_{θ}[l(θ′|x)] = ∫ p(x|θ)l(θ′|x)dx = ∫ p(x|θ) log p(x|θ′)dx
= ∫ p(x|θ) log (p(x|θ)\frac{p(x|θ′)}{p(x|θ)}) dx
= ∫ p(x|θ) log p(x|θ)dx − ∫ p(x|θ) log \frac{p(x|θ)}{p(x|θ)′} dx
= −H(p(x|θ)) − D_{KL}(p(x|θ))||p(x|θ′))  (B.235)

where H(p(x|θ)) is the entropy of p(x|θ)), which is independent of the estimated parameter θ)′, and D_{KL}(p(x|θ))||p(x|θ′)) is the KL-divergence between p(x|θ)) and p(x|θ′)), which is zero when the estimated parameter becomes the same as the true parameter θ′ = θ.

Now we see that the optimal model parameter that maximizes the likelihood is also the one that minimizes the KL-divergence:

θ_{MLE} = arg max_{θ′} L(θ′|x) = arg min_{θ′} D_{KL}(p(x|θ))||p(x|θ′))  (B.236)

Consider the simple example of the MLE method applied to estimate the two parameters in θ = {m, Σ} of the multivariate Gaussian distribution

N (x, m, Σ) = \frac{1}{(2π)^{n/2} |Σ|^{1/2}} exp [−\frac{1}{2}(x − m)^{T} Σ^{−1}(x − m)]  (B.237)

Given an observed dataset containing N i.i.d. data samples X = {x_{1}, ⋯, x_{N}}, we can get the likelihood function

L(θ|X) = L(m, Σ|X) = p(X|m, Σ) = ∏_{i=1}^{N} N (x_{i}|m, Σ)  (B.238)

and the log-likelihood function:

l(m, Σ|X) = log L(m, Σ|X) = log [∏_{i=1}^{N} N (x_{i}|m, Σ)] = ∑_{i=1}^{N} log N (x_{i}|m, Σ)

= ∑_{i=1}^{N} [−\frac{d}{2} log 2π − \frac{1}{2} log |Σ| − \frac{1}{2}(x_{i} − m)^{T} Σ^{−1}(x_{i} − m)] (B.239)

To find the optimal parameters m and Σ that maximize this log-likelihood, we set its derivative with respect to m and Σ to zero and solve the resulting equations:

• Find m

\frac{∂}{∂m}l(m, Σ|X) = −\frac{1}{2} ∑_{i=1}^{N} \frac{∂}{∂m} ((x_{i} − m)^{T} Σ^{−1}(x_{i} − m))

= −\frac{1}{2} ∑_{i=1}^{N} Σ^{−1}(x_{i} − m) = 0  (B.240)

Pre-multiplying −2Σ on both sides, we get

∑_{i=1}^{N}(x_{i} − m) = 0  (B.241)

solving which we get the optimal estimation of m:

m̂ = \frac{1}{N} ∑_{i=1}^{N} x_{i}  (B.242)

• Find Σ

\frac{∂}{∂Σ}l(m, Σ|X) = −\frac{N}{2} \frac{∂}{∂Σ} log |Σ| − \frac{1}{2} ∑_{i=1}^{N} \frac{∂}{∂Σ} ((x_{i} − m)^{T} Σ^{−1}(x_{i} − m))

= −\frac{N}{2} Σ^{−1} + \frac{1}{2} ∑_{i=1}^{N} Σ^{−1}(x_{i} − m)(x_{i} − m)^{T} Σ^{−1} = 0 (B.243)

Pre and post multiplying Σ on both sides and solving the resulting equation we get the optimal estimation of Σ:

Σ̂ = \frac{1}{N} ∑_{i=1}^{N}(x_{i} − m)(x_{i} − m)^{T}  (B.244)

Here we have used the following facts:

\frac{d}{dA} log |A| = (A^{−1})^{T}  (B.245)
\frac{d}{dA} (a^{T} A^{−1}b) = −(A^{−1})^{T} ab^{T} (A^{−1})^{T}  (B.246)

Note that the rank of Σ̂ in Eq. (B.244) is N − 1, representing its degrees of freedom due to the N samples, assumed to be independent, and the extra constraint in Eq. (B.242). When d > N − 1, the matrix Σ̂ does not have a full rank and is therefore non-invertible.

For more details regarding derivatives of a scalar function with respect to

vector and matrix variables, see Matrix Cookbook here: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf.

B.3.4 Natural gradient

The gradient method is commonly used in optimization to minimize or maximize an objective function J(x) in Euclidean space based on the following iteration:

x_{n+1} = x_{n} + Δx^{∗}  (B.247)

where Δx^{∗} is the optimal increment among all those Δx = x_{n+1} − x_{n} within a small neighborhood of the current x_{n}:

Δx^{∗} = arg max_{||Δx||<C} J(x + Δx)  (B.248)

The solution of this constrained optimization problem can be found by maximizing the following Lagrangian function

L(Δx) = L(x + Δx) + μ(||Δx||^{2} − C)  (B.249)

The optimal Δx^{∗} can be found by setting the derivative of L(Δx) to zero

\frac{d}{d Δx}L(x + Δx) = g_{L}(x) + \frac{μ}{2}Δx = 0  (B.250)

and solving the resulting equation to get

Δx^{∗} = −\frac{2}{μ}g_{J}(x)  (B.251)

Now the optimal x^{∗} can be found iteratively

x_{n+1} = x_{n} + Δx_{n} = x_{n} − λg_{J}(x_{n})  (B.252)

where the step size λ = 2/μ can be adjusted as needed.

However, in some cases the function to be optimized is not in the Euclidean space spanned by a set of orthonormal basis vector in which an inner product is defined and the Euclidean distance is used to measure the difference between two points in the space, but in a Riemannian manifold (a smooth manifold locally similar to a Euclidean space), in which a small neighboring area of a point can be approximated by its tangent space spanned by a set of vectors {e_{1}, ⋯, e_{n}} not necessarily orthogonal to each other, and the squared length of an infinitesimal arc dx = [dx_{1}, ⋯, dx_{d}]^{T} on the manifold can be expressed in quadratic form as:

||dx||^{2} = (∑_{i=1}^{n} dx_{i}e_{i})^{T} (∑_{j=1}^{n} dx_{j}e_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{n} dx_{i}dx_{j}e_{i}^{T} e_{j}
= dx^{T} G dx  (B.253)

Here G, called the metric tensor, is a positive-definite matrix based on the inner products of the basis vectors in the tangent space:

G = [ \begin{matrix}e_{1}^{T} e_{1} & ⋯ & e_{1}^{T} e_{n} \\ ⋮ & ... & ⋮ \\ e_{n}^{T} e_{1} & ⋯ & e_{n}^{T} e_{n} \end{matrix} ]  (B.254)

In the special case when the Riemannian space is a Euclidean space spanned by orthonormal basis vectors satisfying e_{i}^{T} e_{j} = δ_{ij}, we have G = I, and the squared arc length above becomes the conventional Euclidean squared norm based on inner product: ||dx||^{2} = dx^{T} dx = ∑_{i} dx_{i}^{2}.

To minimize an objective function J(x) in a Riemannian manifold iteratively, we can still find the optimal increment step Δx as given in Eq. (B.248), that maximizes the Lagrangian in Eq. (B.249), but now the squared norm is given in Eq. (B.253):

L(Δx) = L(x + Δx) + μ(Δx^{T} GΔx − C)  (B.255)

Now we solve the equation

\frac{d}{d Δx}L(x + Δx) = \frac{d}{d Δx}L(x + Δx) + μ\frac{d}{d Δx} (Δx^{T} GΔx − C)
= g_{L}(x) + \frac{μ}{2}GΔx  (B.256)

to get the optimal increment:

Δx^{∗} = −\frac{μ}{2}G^{−1}g_{L}(x) = −\frac{μ}{2}g̃_{L}(x)  (B.257)

where g̃_{L}(x) = G^{−1}g_{L}(x) is the natural gradient, and the iteration becomes

x_{n+1} = x_{n} + Δx_{n} = x_{n} − δg̃_{L}(x)  (B.258)

B.3.5 Fisher Information and Natural Gradient

The Fisher information measures the amount of information carried by an observed random x about the unknown parameter θ of a model distribution p(x|θ) of x.

The likelihood function of the parameter θ for a model p(x|θ) given observed data x is

L(θ|x) ∝ p(x|θ)  (B.259)

and the log-likelihood is

l(θ|x) = log L(θ|x) = log p(x|θ)  (B.260)

The gradient of the log-likelihood is

g_{l}(θ) = \frac{d}{dθ} log p(x|θ) = \frac{d p(x|θ)/dθ}{p(x|θ)} = \frac{g_{p}(θ)}{p(x|θ)}  (B.261)

where g_{p}(θ) = d p(x|θ)/dθ is the gradient of p(x|θ). The expectation of g_{l} is zero:

E_{θ}[g_{l}(θ)] = ∫ (\frac{g_{p}(θ)}{p(x|θ)}) p(x|θ)dx = ∫ g_{p}(θ)dx
= ∫ \frac{d}{dθ} p(x|θ)dx = \frac{d}{dθ} ∫ p(x|θ)dx = \frac{d}{dθ} 1 = 0 (B.262)

and the covariance of g_{l} is defined as the Fisher information:

F(θ) = E_{θ}[g_{l}(θ)g_{l}^{T} (θ)]  (B.263)

which can be estimated based on an observed dataset x = [x_{1}, ⋯, x_{N}]:

F(θ) = \frac{1}{N} ∑_{n=1}^{N} g_{l}(θ)g_{l}^{T} (θ)  (B.264)

The Hessian of the log-likelihood l(θ|x) is:

H_{l}(θ) = \frac{d^{2}}{dθ^{2}} l(θ|x) = \frac{d}{dθ} g_{l}(θ) = \frac{d}{dθ} [\frac{g_{p}(θ)}{p(x|θ)}]
= \frac{H_{p}(θ)p(x|θ) − g_{p}(θ)g_{p}^{T} (θ)}{p^{2}(x|θ)} = \frac{H_{p}(θ)}{p(x|θ)} − g_{l}(θ)g_{l}^{T} (θ) (B.265)

The expectation of H_{l} is

E_{θ}[H_{l}(θ)] = E_{θ} [\frac{H_{p}(θ)}{p(x|θ)}] − E_{θ} [g_{l}(θ)g_{l}^{T} (θ)]  (B.266)

The first term is

E_{θ} [\frac{H_{p}(θ)}{p(x|θ)}] = ∫ [\frac{H_{p}(θ)}{p(x|θ)}] p(x|θ)dx = ∫ H_{p}(θ)dx = ∫ \frac{d^{2}}{dθ^{2}} p(x|θ)dx
= \frac{d^{2}}{dθ^{2}} ∫ p(x|θ)dx = \frac{d^{2}}{dθ^{2}} 1 = 0  (B.267)

and the second term is the Fisher information defined in Eq. (B.263), and we have:

F(θ) = −E_{θ}[H_{l}(θ)] = E_{θ} [g_{l}(θ)g_{l}^{T} (θ)]  (B.268)

of which the component in the ith row and jth column is expected product of the ith and jth components of g_{l}(θ):

F_{ij} = E[g_{i} g_{j}] = E [\frac{∂ log p(x|θ)}{∂θ_{i}} \frac{∂ log p(x|θ)}{∂θ_{j}}]  (B.269)

We note that g_{i} = ∂ log p(x|θ)/∂θ_{i} represents how much variation in the pdf p(x|θ) of x will be caused by parameter θ_{i}, or how much information about θ_{i} is carried by x. If in the extreme case when g_{i} = 0, then x carries no information about θ_{i} as it is irrelevant to θ_{i}. Therefore the Fisher information containing all F_{ij} = E[g_{i} g_{j}] represents the amount of information about θ carried by x.

While building a statistical model p(x|θ) of the observed dataset, we need to approximate the unknown parameter θ by the estimated θ′ based on the samples of variable x, so that the KL-divergence between p(x|θ′) and the true p(x|θ) (Eq. (B.198)) is minimized:

D_{KL}(p(x|θ)||p(x|θ′)) = E_{θ}[log p(x|θ) − log p(x|θ′)]  (B.270)

Consider its first and second order derivatives with respect to θ′:

• The gradient:

g_{KL}(θ′) = E_{θ} [\frac{d}{dθ′} [log p(x|θ) − log p(x|θ′)]]
= −E_{θ} [\frac{d}{dθ′} log p(x|θ′)] = −E_{θ}[g_{l}(θ′)]  (B.271)

• The Hessian:

H_{KL}(θ′) = −E_{θ} [\frac{d}{dθ′} g_{l}(θ′)] = −E_{θ}[H_{l}(θ′)]  (B.272)

Evaluating both g_{KL}(θ′) and H_{KL}(θ′) at θ′ = θ, we get (Eqs. (B.262) and (B.268)):

g_{KL}(θ) = −E_{θ}[g_{l}(θ)] = 0,  and H_{KL}(θ) = F(θ)  (B.273)

Now the KL-divergence above can be approximated by the first three terms of its Taylor expansion in the neighborhood of θ in terms of Δθ = θ′ − θ:

D_{KL}(p(x|θ)||p(x|θ′)) ≈ D_{KL}(p(x|θ)||p(x|θ)) + g_{KL}^{T}(θ)Δθ
+ \frac{1}{2}Δθ^{T} H_{KL}(θ)Δθ = \frac{1}{2}Δθ^{T} F(θ)Δθ (B.274)

Comparing this with Eq. (B.253), we see that we can treat each component g_{i} = ∂ log p(x|θ)/∂θ_{j} of g_{l}(θ) as a basis vector in the Riemanniian space for all distributions p(x|θ), then the KL-divergence above becomes the squared distance between the two distributions:

D_{KL}(p(x|θ)||p(x|θ′)) ≈ \frac{1}{2}Δθ^{T} F(θ)Δθ = \frac{1}{2} ∑_{i} ∑_{j} Δθ_{i}Δθ_{j}g_{i}g_{j}  (B.275)

Now the optimal model parameter θ that maximizes the likelihood or equivalently minimizes the KL-divergence (Eq. (B.236)) can be found iteratively by the natural gradient method (Eq (B.258)):

θ_{n+1} = θ_{n} + Δθ_{n} = θ_{n} + δg̃_{L}(θ_{n})  (B.276)

B.3.6 Bias-Variance Trade-off

The mean squared error as defined below is a common way to measure the error of the estimation of some parameter, or, more generally, the performance of a supervised learning algorithm (for either regression or classification). Let θ̂ be an estimator of the parameter θ of a data model. Then mean squared error (MSE) of this estimator is

MSE(θ̂) = E[(θ̂ − θ)^{2}]
= E[(θ̂ − E[θ̂] + E]θ̂] − θ)^{2}]
= E[(θ̂ − E[θ̂])^{2}] + 2 E[(θ̂ − E[θ̂])(E[θ̂] − θ)] + E[(E[θ̂] − θ)^{2}]
= E[(θ̂ − E[θ̂])^{2}] + E[(E[θ̂] − θ)^{2}]
= Var(θ̂) + Bias(θ̂)  (B.277)

where the middle term is zero as E[θ̂ − E[θ̂]] = E[θ̂] − E[θ̂] = 0.

The equation above indicates that the MSE is composed of two parts:

• variance error Var(θ̂) = E[(θ̂ − E[θ̂])^{2}]:

caused by the oversensitivity of a learning algorithm to small fluctuations in the training dataset due to random noise rather than the intended property of the data, a problem called overfitting.

• bias error Bias(θ̂) = E[(E[θ̂] − θ)^{2}]:

caused by the under-sensitivity of a learning algorithm to the dominating variation in the training dataset representing the essential relationship in the training dataset, a problem called underfitting.

In general, a proper bias-variance trade-off needs to be made for an algorithm to avoid either overfitting or underfitting, so that it can learn the essential relationship in the data while not affected by the inevitable noise in the data.

Notes

Chapter 2

1 Lewis Fry Richardson (1881–1953).