Non-Gaussianity is Independence

The theoretical foundation of ICA is the Central Limit Theorem, which states that the distribution of the sum (average or linear combination) of $N$ independent random variables approaches Gaussian as $N\rightarrow \infty$. For example, the face value of a dice has a uniform distribution from 1 to 6. But the distribution of the sum of a pair of dice is no longer uniform. It has a maximum probability at the mean of 7. The distribution of the sum of the face values will be better approximated by a Gaussian as the number of dice increases.

Specifically, if $x_i$ are random variables independently drawn from an arbitrary distribution with mean $\mu$ and variance $\sigma^2$. Then the distribution of the mean $x=\sum_{i=1}^N x_i/N$ approaches Gaussian with mean $\mu$ and variance $\sigma^2/N$.

To solve the BSS problem, we want to find a matrix ${\mathbf W}$ so that ${\mathbf y=Wx=WAs \approx s}$ is as close to the independent sources ${\mathbf s}$ as possible. This can be seen as the reverse process of the central limit theorem above.

Consider one component $y_i={\mathbf w_i^TAs}$ of ${\mathbf y}$, where ${\mathbf w_i^T}$ is the ith row of ${\mathbf W}$. As a linear combination of all components of ${\mathbf s}$, $y_i$ is necessarily more Gaussian than any of the components unless it is equal to one of them (i.e., ${\mathbf w_i^TA}$ has only one non-zero component. In other words, the goal ${\mathbf y \approx s}$ can be achieved by finding ${\mathbf W}$ that maximizes the non-Gaussianity of ${\mathbf y=Wx=WAs}$ (so that ${\mathbf y}$ is least Gaussian). This is the essence of all ICA methods. Obviously if all source variables are Gaussian, the ICA method will not work.

Based on the above discussion, we get requirements and constrains for the ICA methods:

• The number of observed variables $m$ must be no fewer than the number of independent sources $n$ (assume $m=n$ in the following).
• The source components are stochastically independent, and have to be non-Gaussian (with possible exception of at most one Gaussian).
• The estimation of $A$ and $S$ is up to a scaling factor $c_i$. Let ${\mathbf C}=diag(c_1,\cdots,c_n)$ and ${\mathbf C}^{-1}= diag(1/c_1,\cdots,1/c_n)$, and ${\mathbf A}'={\mathbf AC}^{-1}$ and ${\mathbf s'=Cs}$, we have
  
  $${\mathbf x=As=[AC^{-1}][Cs]=A's'}$$
  
  
  Also the scaling factor $c_i$ could be either positive or negative. For this reason, we will always assume the independent components have unit variance $E\{s_i^2\}=1$. As they are also uncorrelated (all independent variables are uncorreclated), we have $E\{s_is_j\}=\delta_{ij}$, i.e.,
  
  $$E\{SS^T\}=I$$
  
  

• The estimated independent components are not in any particular order. When the order of the corresponding elements in both ${\mathbf s}$ and ${\mathbf A}$ is rearranged, ${\mathbf x=As}$ still holds.

All ICA methods are based on the same fundamental approach of finding a matrix ${\mathbf W}$ that maximizes the non-Gaussianity of ${\mathbf s=Wx}$ thereby minimizing the independence of $S$, and they can be formulated as:

$${\bf \mbox{ICA method}=\mbox{objective function}+\mbox{optimization algorithm} }$$

All ICA methods are an optimization process (always iterative) to find a matrix ${\mathbf W}$ to maximize some objective function that measures the degree of non-Gaussianity or independence of the estimated components ${\mathbf s=Wx}$. In the following, we will discuss some common objective functions.