The Principal Component Transform

The Principal Component Transform is also called Karhunen-Loeve Transform (KLT), Hotelling Transform, or Eigenvector Transform.

Let $\phi_k$ and $\lambda_k$ be the kth eigenvector and eigenvalue of the covariance matrix $\Sigma_X$:

$$\Sigma_X \phi_k=\lambda_k\phi_k\;\;\;\;\;\;(k=0,\cdots,N-1)$$

or in matrix form:

$$\left[ \begin{array}{ccc}\cdots &\cdots &\cdots \\ \cdots &... ...a_k\left[ \begin{array}{c} \ \phi_k \ \ \end{array} \right]$$

We can construct an $N \times N$ matrix $\Phi$

$$\Phi\stackrel{\triangle}{=}[\phi_0, \cdots, \phi_{N-1}]$$

Since the columns of $\Phi$ are the eigenvectors of a symmetric (Hermitian if $X$ is complex) matrix $\Sigma_X$, $\Phi$ is orthogonal (unitary):

$$\Phi^{T} \Phi = I,\;\;\;\;\mbox{i.e.,}\;\;\;\; \Phi^{-1}=\Phi^{T}$$

and we have

$$\Sigma_X\Phi=\Phi \Lambda$$

or in matrix form:

$$\left[ \begin{array}{ccc}\cdots &\cdots &\cdots \\ \cdots &... ... &\cdots \ \cdots &\cdots &\lambda_{N-1} \end{array} \right]$$

where $\Lambda=diag(\lambda_0, \cdots, \lambda_{N-1} )$. Or, we have

$$\Phi^{-1}\Sigma_X\Phi=\Phi^{T} \Sigma_X \Phi = \Phi^{-1}\Phi \Lambda=\Lambda$$

We can now define the orthogonal (unitary if $X$ is complex) Principal Component Transform of $X$. The forward transform:

$$Y=\left[ \begin{array}{l} y_0 y_1 \cdots y_{N-1} \end... ...T_0 \phi^T_1 \cdots \phi^T_{N-1} \end{array} \right] X$$

and the inverse transform

$$X=\Phi Y=[ \phi_0, \phi_1, \cdots, \phi_{N-1} ] \left[ \begin{array}{l} y_0 y_1 \cdots y_{N-1} \end{array} \right]$$

The ith component of the forward transform $Y=\Phi^{T} X$ is the projection of $X$ on $\phi_i$:

$$y_i=(\phi_i,X)=\phi_i^TX$$

and the inverse transform $X=\Phi Y$ represents $X$ in the N-dimensional space spanned by $\phi_i$ $(i=0, 1, \cdots, N-1)$:

$$X=\sum_{i=0}^{N-1} y_i \phi_i$$