Probability of Random Vectors

• Multiple Random Variables

  Each outcome of a random experiment may need to be described by a set of $N>1$ random variables $\{x_1, \cdots, x_N \}$, or in vector form:
  

  $${\bf x}=[x_1,\; \cdots \;,x_N]^T$$
  
  
  which is called a random vector. In signal processing ${\bf x}$ is often used to represent a set of $N$ samples $x_i=x(t_i)$ of a random signal $x(t)$ (a random process).

• Joint Distribution Function and Density Function

  The joint distribution function of a random vector ${\bf x}$ is defined as
  

  $$F_X(u_1, \cdots, u_N) = P(x_1<u_1,\cdots, x_N<u_N)$$

  $$= \int_{-\infty}^{u_1} \cdots \int_{-\infty}^{u_N} p(\xi_1,\cdots,\xi_N) d\xi_1 \cdots d\xi_N$$

  
  
  where $p(\xi_1,\cdots,\xi_N)$ is the joint density function of the random vector ${\bf x}$.

• Independent Variables

  For convenience, let us first consider two of the $N$ variables and rename them as $x$ and $y$. These two variables are independent iff
  

  $$P(A\cap B)=P(x<u, y<v)=P(x<u) P(y<v)=P(A)P(B)$$
  
  
  where events $A$ and $B$ are defined as ``$x<u$'' and ``$y<v$'', respectively. This definition is equivalent to
  
  $$p(x,y)=p(x)p(y)$$
  
  
  as this will lead to
  

  $$P(x<u,y<v) = \int_{-\infty}^u\int_{-\infty}^v p(\xi,\eta)d\xi d\eta = \int_{-\infty}^u\int_{-\infty}^v p(\xi)p(\eta)d\xi d\eta$$

  $$= \int_{-\infty}^u p(\xi)d\xi \int_{-\infty}^v p(\eta)d\eta = P(x<u)P(y<v)$$

  
  

  Similarly, a set of $N$ variables are independent iff
  

  $$p(x_1,\cdots,x_N)=p(x_1) p(x_2)\cdots p(x_N)$$
  
  

• Mean Vector

  The expectation or mean of random variable $x_i$ is defined as
  

  $$\mu_i=E(x_i)\stackrel{\triangle}{=} \int_{-\infty}^{\infty} ... ...}^{\infty} \xi_i p(\xi_1,\cdots,\xi_N) d\xi_1\cdots d\xi_N$$
  
  

  The mean vector of random vector ${\bf x}$ is defined as
  

  $${\bf m}=E({\bf x})\stackrel{\triangle}{=} [ E(x_1), \cdots, E(x_N) ]^T=[\mu_1, \cdots, \mu_N]^T$$
  
  
  which can be interpreted as the center of gravity of an N-dimensional object with $p(x_1,\cdots,x_N)$ being the density function.

• Covariance Matrix

  The variance of random variable $x_i$ is defined as
  

  $$\sigma_i^2 \stackrel{\triangle}{=} E[(x_i-\mu_i)^2] = E(x_i^2)-\mu_i^2$$

  $$= \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} (\xi_i-\mu_i)^2 p(\xi_1,\cdots,\xi_N) d\xi_1\cdots d\xi_N$$

  
  

  The covariance of $x_i$ and $x_j$ is defined as
  

  $$\sigma_{ij}^2=Cov(x_i,x_j) \stackrel{\triangle}{=} E[(x_i-\mu_i)(x_j-\mu_j)] =E(x_ix_j)-\mu_i\mu_j$$

  $$= \int_{-\infty}^{\infty} \cdots \int_{-\infty}^{\infty} \xi_i\xi_j p(\xi_1,\cdots,\xi_N) d\xi_1\cdots d\xi_N -\mu_i\mu_j$$

  
  

  The covariance matrix of a random vector ${\bf x}$ is defined as
  

  $${\bf\Sigma} & = & E[({\bf x}-{\bf m})({\bf x}-{\bf m})^T]=E... ...{ij}^2 & .. \\ .. & .. & .. \end{array} \right]_{N\times N}$$
  
  
  where
  
  $$\sigma_{ij}^2=E(x_ix_j)-\mu_i\mu_j$$
  
  
  is the covariance of $x_i$ and $x_j$. When $i=j$, $\sigma_i^2=E(x_i^2)-\mu_i^2$ is the variance of $x_i$, which can be interpreted as the amount of information, or energy, contained in the ith component of the signal ${\bf x}$. And the total information or energy contained in ${\bf x}$ is represented by
  
  $$tr\;{\bf\Sigma}=\sum_{i=1}^N \sigma_i^2$$
  
  

  $\Sigma$ is symmetric as $\sigma_{ij}^2=\sigma_{ji}^2$. Moreover, it can be shown that $\Sigma$ is also positive definite, i.e., all its eigenvalues are greater than zero $\lambda_i > 0$ for all $i=1,\cdots,N$, and we have
  

  $$tr\; {\bf\Sigma}=\sum_{i=1}^N \lambda_i > 0$$
  
  
  and
  
  $$det\; {\bf\Sigma}=\prod_{i=1}^N \lambda_i > 0$$
  
  

  Two variables $x_i$ and $x_j$ are uncorrelated iff $\sigma_{ij}^2=0$, i.e.,
  

  $$E(x_ix_j)=E(x_i)E(x_j)=\mu_i\mu_j$$
  
  

  If this is true for all $i \ne j$, then ${\bf x}$ is called uncorrelated or decorrelated and its covariance matrix ${\bf\Sigma}$ becomes a diagonal matrix with only non-zero $\sigma_i^2$ $(i=1,\cdots, N)$ on its diagonal.

  If $x_i\;\;(i=1,\cdots,N)$ are independent, $p(x_1,\cdots,x_N)=p(x_1)\cdots p(x_N)$, then it is easy to show that they are also uncorrelated. However, uncorrelated variables are not necessarily independent. (But uncorrelated variables with normal distribution are also independent.)

• Autocorrelation Matrix

  The autocorrelation matrix of $X$ is defined as
  

  $${\bf R}\stackrel{\triangle}{=}E({\bf xx}^T)= \left[ \begin... ...ts & \cdots & \cdots \end{array} \right]_{N\times N} \nonumber$$
  
  
  where
  
  $$r_{ij}\stackrel{\triangle}{=}E(x_ix_j)=\sigma_{ij}^2+\mu_i \mu_j$$
  
  
  Obviously $R$ is symmetric and we have
  
  $${\bf\Sigma}={\bf R}-{\bf mm}^T$$
  
  
  When ${\bf m}=0$, we have ${\bf\Sigma}={\bf R}$.

  Two variable $x_i$ and $x_j$ are orthogonal iff $r_{ij}=0$. Zero mean random variables which are uncorrelated are also orthogonal.

• Mean and Covariance under Unitary Transforms

  A unitary (orthogonal) transform of ${\bf x}$ is defined as
  

  $$\left\{ \begin{array}{c} {\bf y}={\bf A}^T{\bf x} {\bf x}={\bf A}{\bf y} \end{array} \right.$$
  
  
  where ${\bf A}$ is a unitary (orthogonal) matrix
  
  $${\bf A}^{*T}={\bf A}^{-1}$$
  
  
  and ${\bf y}$ is another random vector.

  The mean vector ${\bf m}_Y$ and the covariance matrix ${\bf\Sigma}_Y$ of ${\bf y}$ are related to the ${\bf m}_X$ and ${\bf\Sigma}_X$ of ${\bf x}$ as shown below:
  

  $${\bf m}_Y=E({\bf y})=E({\bf A}^T{\bf x})={\bf A}^TE({\bf x})={\bf A}^T{\bf m}_X$$
  
  
  
  

  $${\bf\Sigma}_y = E({\bf yy}^T)-{\bf m}_y{\bf m}_y^T =E({\bf A}^T{\bf xx}^T{\bf A})-{\bf A}^T{\bf m}_x{\bf m}_x^T{\bf A}$$

  $$= {\bf A}^TE({\bf xx}^T){\bf A}-{\bf A}^T{\bf m}_x{\bf m}_x^T{\bf A... ...f A}^T[E({\bf xx}^T)-{\bf m}_x{\bf m}_x^T]{\bf A}={\bf A}^T{\bf\Sigma}_x{\bf A}$$

  
  

  Unitary transform does not change the trace of $\Sigma$:
  

  $$tr\; {\bf\Sigma}_y = tr\;[E({\bf yy}^T)-{\bf m}_y{\bf m}_y^T] =E[tr\;({\bf yy}^T)]-tr\;({\bf m}_y{\bf m}_y^T)$$

  $$= E({\bf y}^T{\bf y})-{\bf m}_y^T{\bf m}_y =E({\bf x}^T{\bf AA}^T{\bf x})-{\bf m}_x^T{\bf AA}^T{\bf m}_x$$

  $$= E({\bf x}^T{\bf x})-{\bf m}_x^T{\bf m}_x=tr\;{\bf\Sigma}_x$$

  
  
  which means the total amount of energy or information contained in ${\bf x}$ is not changed after a unitary transform ${\bf y}={\bf A}^T{\bf x}$ (although its distribution among the $N$ components is changed).

• Normal Distribution

  The density function of a normally distributed random vector $X$ is:
  

  $$p(x_1,\cdots, x_N)=N({\bf x}, {\bf m}, {\bf\Sigma}) =\frac{1... ...2}({\bf x}-{\bf m})^T{\bf\Sigma}^{-1}({\bf x}-{\bf m}) \right]$$
  
  
  where ${\bf m}$ and ${\bf\Sigma}$ are the mean vector and covariance matrix of ${\bf x}$, respectively. When $N=1$, ${\bf\Sigma}$ and ${\bf m}$ become $\sigma$ and $\mu$, respectively, and the density function becomes single variable normal distribution.

  To find the shape of a normal distribution, consider the iso-value hyper surface in the N-dimensional space determined by equation
  

  $$N({\bf x},{\bf m},{\bf\Sigma})=c_0$$
  
  
  where $c_0$ is a constant. This equation can be written as
  
  $$({\bf x}-{\bf m})^T {\bf\Sigma}^{-1} ({\bf x}-{\bf m}) = c_1$$
  
  
  where $c_1$ is another constant related to $c_0$, ${\bf m}$ and ${\bf\Sigma}$. For $N=2$ variables $x$ and $y$, we have
  

  $$(X-M)^T \Sigma^{-1} ({\bf x}-{\bf m})=[x-\mu_x, y-\mu_y] \left[ ... ...rray} \right] \left[ \begin{array}{c} x-\mu_x y-\mu_y \end{array} \right]$$

  $$= A(x-\mu_x)^2+B(x-\mu_x)(y-\mu_y)+C(y-\mu_y)^2$$

  $$= Ax^2+Bxy+Cy^2-(2A\mu_x+B\mu_y)x-(2C\mu_y+B\mu_x)y +(A\mu_x^2+B\mu_x\mu_y+C\mu_y^2)=c_1$$

  
  
  Here we have assumed
  
  $${\bf\Sigma}^{-1} =\left[ \begin{array}{cc} A & B/2 B/2 & C \end{array} \right]$$
  
  

  The above quadratic equation represents an ellipse (instead of any other quadratic curve) centered at ${\bf m}=[\mu_1, \mu_2]^T$, because $\Sigma^{-1}$, as well as ${\bf\Sigma}$, is positive definite:
  

  $$\left\vert {\bf\Sigma}^{-1} \right\vert = AC-B^2/4 > 0,\;\;\;\;\;\mbox{i.e.}\;\;\;\;\; \Delta=B^2-4AC<0$$
  
  

  Recall in general that the discriminant $\Delta=B^2-4AC$ of a quadratic equation
  

  $$f(x,y)=Ax^2+Bxy+Cy^2+Dx+Ey+F=0$$
  
  
  determines the shpae of the corresponding conic section in a plane:
  
  $$f(x,y)\;\mbox{ is } \left\{\begin{array}{ll} \mbox{an ellips... ...=0 \\ \mbox{a hyperbola}&\mbox{if }\Delta>0\end{array}\right.$$
  
  

  When $N>2$, the equation $N({\bf x},{\bf m},{\bf\Sigma})=c_0$ represents a hyper ellipsoid in the N-dimensional space. The center and spatial distribution of this ellipsoid are determined by ${\bf m}$ and ${\bf\Sigma}$, respectively.

  In particular, when ${\bf x}=[x_1,\cdots,x_N]^T$ is decorrelated, i.e., $\sigma_{ij}=0\;\;for\;all\;\;i \neq j$, ${\bf\Sigma}$ becomes a diagonal matrix
  

  $${\bf\Sigma}=diag[\sigma_1^2,\cdots,\sigma_N^2] =\left[ \be... ...dots & \vdots \\ 0 & \cdots & \sigma_N^2 \end{array} \right]$$
  
  
  and equation $N({\bf x},{\bf m},{\bf\Sigma})=c_0$ can be written as
  
  $$({\bf x}-{\bf m})^T{\bf\Sigma}^{-1}({\bf x}-{\bf m}) =\sum_{i=1}^N \frac{(x_i-\mu_i)^2}{\sigma_i^2} = c_1$$
  
  
  which represents a standard ellipsoid with all its axes parallel to those of the coordinate system.

• Estimation of ${\bf m}$ and ${\bf\Sigma}$

  When $p(x_1,\cdots,x_N)$ is not known, ${\bf m}$ and ${\bf\Sigma}$ cannot be found by their definitions. However, they can be estimated if a large number of outcomes ( ${\bf x}_j,\;\;j=1,\cdots,K$) of the random experiment in question can be observed.

  The mean vector ${\bf m}$ can be estimated as
  

  $$\hat{{\bf m}}=\frac{1}{K}\sum_{j=1}^K X_j$$
  
  
  i.e., the ith element of ${\bf m}$ is estimated as
  
  $$\hat{\mu}_i=\frac{1}{K}\sum_{j=1}^K x_i^{(j)}$$
  
  
  where $x_i^{(j)}$ is the ith element of ${\bf x}_j$.

  The autocorrelation ${\bf R}$ can be estimated as
  

  $$\hat{{\bf R}}= \frac{1}{K}\sum_{j=1}^K {\bf x}_j {\bf x}_j^T$$
  
  

  And the covariance matrix ${\bf\Sigma}$ can be estimated as
  

  $$\hat{{\bf\Sigma}} = \frac{1}{K}\sum_{j=1}^K {\bf x}_j {\bf x... ...m}}\hat{{\bf m}}^T=\hat{{\bf R}}-\hat{{\bf m}}\hat{{\bf m}}^T$$