Preprocessing for ICA

To simplify the ICA algorithms, the following preprocessing steps are usually taken:

• Centering

  Subtract the mean $E\{{\bf x}\}$ from the observed variable ${\bf x=As}$ so it has zero mean. By doing so, the sources ${\bf s}$ also become zero mean because $E\{{\bf x}\}={\bf A}\;E\{{\bf s}\}=0$. When the mixing matrix ${\bf A}$ is available, $E\{{\bf s}\}$ can be estimated to be ${\bf A}^{-1}E\{{\bf x}\}$.

• Whitening

  Transform observed variables $X$ so that they are uncorrelated and have unit variance. We first obtain the eigenvalues $\lambda_i$ and their corresponding eigenvectors ${\bf\phi}_i$ of the covariance matrix $E\{{\bf xx}^T\}$, and form the diagonal eigenvalue matrix ${\bf\Lambda}=diag(\lambda_1, \cdots,\lambda_m)$ and orthogonal eigenvector matrix ${\bf\Phi}= [{\bf\phi}_1,\cdots,{\bf\phi}_m]$ ( ${\bf\Phi}^{-1}={\bf\Phi}^T$). We have

  $$E\{{\bf xx}^T\}{\bf\Phi}={\bf\Phi\Lambda}, \;\;\; \;\;\;\; {\bf\Lambda}^{-1/2}{\bf\Phi}^T E\{{\bf xx}^T\} {\bf\Phi \Lambda^{-1/2}}={\bf I}$$ (228)

  If we carry out a linear transform ${\bf T}={\bf\Lambda}^{-1/2} {\bf\Phi}^T$ so that ${\bf x'=TX}=({\bf\Lambda}^{-1/2}{\bf\Phi}^T) X$, the covariance matrix of ${\bf x}'$ becomes
  

  $$E\{{\bf x'x'}^T\} = E\{({\bf\Lambda}^{-1/2}{\bf\Phi}^T {\bf x}) ({\bf\Lambda}^{-1/2}{... ...})^T\} =E\{{\bf\Lambda}^{-1/2}{\bf\Phi}^T {\bf xx}^T {\bf\Phi \Lambda}^{-1/2}\}$$

  $$= {\bf\Lambda}^{-1/2}{\bf\Phi}^T E\{{\bf xx}^T\} {\bf\Phi \Lambda}^{-1/2}={\bf I }$$ (229)

  
  After the transform ${\bf T}={\bf\Lambda}^{-1/2} {\bf\Phi}^T$, the mixing process becomes

  $${\bf x'=Tx=TAs=A's}$$ (230)

  Here the new mixing matrix ${\bf A'=TA}$ is orthogonal, as it satisfies:

  $$E\{{\bf x'x'}^T\}={\bf I}={\bf A}'E\{{\bf ss}^T\} {\bf A'}^T={\bf A'A'}^T$$ (231)

  (recall that we assume $E\{{\bf ss}^T\}={\bf I}$). Similarly we also know that the matrix ${\bf W}$ we seek is also orthogonal, as

  $$E\{{\bf ss}^T\}={\bf I}={\bf W}E\{{\bf x'x'}^T\} {\bf W}^T={\bf WW}^T$$ (232)

  The whitening process reduces the independent variables, the $n^2$ components of the mixing matrix $A'$ to half ($n(n-1)/2$) due to the constraint that ${\bf A}'$ is orthogonal. Moreover, the whitening can also reduce the dimensionality of the problem by ignoring the components corresponding to very small eigenvalues (PCA).