Marginal and conditional distributions of multivariate normal distribution

Assume an n-dimensional random vector

$${\bf x}=\left[\begin{array}{c}{\bf x}_1 {\bf x}_2\end{array}\right]$$

has a normal distribution $N({\bf x},\mu,\Sigma)$ with

$$\mu=\left[\begin{array}{c}\mu_1 \mu_2\end{array}\right]\;\... ...1} & \Sigma_{12}\\ \Sigma_{21}&\Sigma_{22}\end{array}\right]$$

where ${\bf x}_1$ and ${\bf x}_2$ are two subvectors of respective dimensions $p$ and $q$ with $p+q=n$. Note that $\Sigma=\Sigma^T$, and $\Sigma_{21}=\Sigma_{21}^T$.

Theorem 4:

Part a The marginal distributions of ${\bf x}_1$ and ${\bf x}_2$ are also normal with mean vector $\mu_i$ and covariance matrix $\Sigma_{ii}$ ($i=1,2$), respectively.

Part b The conditional distribution of ${\bf x}_i$ given ${\bf x}_j$ is also normal with mean vector

$$\mu_{i\vert j}=\mu_i+\Sigma_{ij}\Sigma_{jj}^{-1}({\bf x}_j-\mu_j)$$

and covariance matrix

$$\Sigma_{i\vert j}=\Sigma_{jj}-\Sigma_{ij}^T\Sigma_{ii}^{-1}\Sigma_{ij}$$

Proof: The joint density of ${\bf x}$ is:

$$f({\bf x})=f({\bf x}_1,{\bf x}_2)=\frac{1}{(2\pi)^{n/2\vert\S... ...vert\Sigma\vert^{1/2}}}exp[-\frac{1}{2}Q({\bf x}_1,{\bf x}_2)]$$

where $Q$ is defined as

$$Q({\bf x}_1,{\bf x}_2) = ({\bf x}-\mu)^T\Sigma^{-1}({\bf x}-\mu)$$

$$= [({\bf x}_1-\mu_1)^T, ({\bf x}-\mu_2)^T] \left[\begin{array}{cc}\... ...ght] \left[\begin{array}{c}{\bf x}_1-\mu_1 {\bf x}_2-\mu_2\end{array}\right]$$

$$= ({\bf x}_1-\mu_1)^T\Sigma^{11}({\bf x}_1-\mu_1)+ 2({\bf x}_1-\mu_1)^T\Sigma^{12}({\bf x}_2-\mu_2)+ ({\bf x}_2-\mu_2)^T\Sigma^{22}({\bf x}_2-\mu_2)$$

Here we have assumed

$$\Sigma^{-1}=\left[\begin{array}{cc}\Sigma_{11} & \Sigma_{12}\... ...} & \Sigma^{12}\\ \Sigma^{21}&\Sigma^{22}\end{array}\right]$$

According to theorem 2, we have

$$\Sigma^{11}=(\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{1... ...\Sigma_{11}^{-1}\Sigma_{12})^{-1}\Sigma_{12}^T\Sigma_{11}^{-1}$$

$$\Sigma^{22}=(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_... ...\Sigma_{22}^{-1}\Sigma_{12}^T)^{-1}\Sigma_{12}\Sigma_{22}^{-1}$$

$$\Sigma^{12}=-\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12})^{-1}=(\Sigma^{21})^T$$

Substituting the second expression for $\Sigma^{11}$, first expression for $\Sigma^{22}$, and $\Sigma^{12}$ into $Q({\bf x}_1,{\bf x}_2)$ to get:

$$Q({\bf x}_1,{\bf x}_2) = ({\bf x}_1-\mu_1)^T [\Sigma_{11}^{-1}+\Sigma_{11}^{-1}\Sigma_{12}... ...igma_{11}^{-1}\Sigma_{12})^{-1}\Sigma_{12}^T\Sigma_{11}^{-1}] ({\bf x}_1-\mu_1)$$

$$-2({\bf x}_1-\mu_1)^T [\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12})^{-1}]({\bf x}_2-\mu_2)$$

$$+({\bf x}_2-\mu_2)^T [(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12})^{-1}] ({\bf x}_2-\mu_2)$$

$$= ({\bf x}_1-\mu_1)^T \Sigma_{11}^{-1} ({\bf x}_1-\mu_1)$$

$$+({\bf x}_1-\mu_1)^T \Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22}-A^T_... ...igma_{11}^{-1}\Sigma_{12})^{-1}\Sigma_{12}^T\Sigma_{11}^{-1}] ({\bf x}_1-\mu_1)$$

$$-2({\bf x}_1-\mu_1)^T [\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12})^{-1}]({\bf x}_2-\mu_2)$$

$$+({\bf x}_2-\mu_2)^T [(\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12})^{-1}] ({\bf x}_2-\mu_2)$$

$$= ({\bf x}_1-\mu_1)^T \Sigma_{11}^{-1} ({\bf x}_1-\mu_1)$$

$$+[({\bf x}_2-\mu_2)-\Sigma_{12}^T\Sigma_{11}^{-1}({\bf x}_1-\mu_1... ...a_{12})^{-1} [({\bf x}_2-\mu_2)-\Sigma_{12}^T\Sigma_{11}^{-1}({\bf x}_1-\mu_1)]$$

The last equal sign is due to the following equations for any vectors $u$ and $v$ and a symmetric matrix $A=A^T$:

$$u^TAu-2u^TAv+v^TAv=u^T Au-u^TAv-u^TAv+v^TAv$$

$$= u^TA(u-v)-(u-v)^TAv=u^TA(u-v)-v^TA(u-v)$$

$$= (u-v)^TA(u-v)=(v-u)^TA(v-u)$$

We define

$$b\stackrel{\triangle}{=}\mu_2+\Sigma_{12}^T\Sigma_{11}^{-1}({\bf x}_1-\mu_1)$$

$$A\stackrel{\triangle}{=}\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12}$$

and

$$\left\{ \begin{array}{ll} Q_1({\bf x}_1)&\stackrel{\triangle... ...\ &=({\bf x}_2-b)^T A^{-1}({\bf x}_2-b) \end{array} \right.$$

and get

$$Q({\bf x}_1,{\bf x}_2)=Q_1({\bf x}_1)+Q_2({\bf x}_1,{\bf x}_2)$$

Now the joint distribution can be written as:

$$f({\bf x}) = f({\bf x}_1,{\bf x}_2) =\frac{1}{(2\pi)^{n/2\vert\Sigma\vert^{1/2}}}exp[-\frac{1}{2}Q({\bf x}_1,{\bf x}_2)]$$

$$= \frac{1}{(2\pi)^{n/2}\vert\Sigma_{11}\vert^{1/2} \vert\Sigma_{22}... ...T\Sigma_{11}^{-1}\Sigma_{12}\vert^{1/2}}exp[-\frac{1}{2}Q({\bf x}_1,{\bf x}_2)]$$

$$= \frac{1}{(2\pi)^{p/2}\vert\Sigma_{11}\vert^{1/2}}exp[-\frac{1}{2}... ...i)^{q/2}\vert A\vert^{1/2}}exp[-\frac{1}{2}({\bf x}_2-b)^T A^{-1}({\bf x}_2-b)]$$

$$= N({\bf x}_1,\mu_1,\Sigma_{11})\;N({\bf x}_2,b,A)$$

The third equal sign is due to theorem 3:

$$\vert\Sigma\vert=\vert\Sigma_{11}\vert \vert\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12}\vert$$

The marginal distribution of ${\bf x}_1$ is

$$f_1({\bf x}_1)=\int f({\bf x}_1,{\bf x}_2) d{\bf x_2} =\frac{... ...ac{1}{2} ({\bf x}_1-\mu_1)^T\Sigma_{11}^{-1}({\bf x}_1-\mu_1)]$$

and the conditional distribution of ${\bf x}_2$ given ${\bf x}_1$ is

$$f_{2\vert 1}({\bf x}_2\vert{\bf x}_1)=\frac{f({\bf x}_1,{\bf ... ...ert^{1/2}}exp[-\frac{1}{2}({\bf x}_2-b)^T A^{-1}({\bf x}_2-b)]$$

with

$$b=\mu_2+\Sigma_{12}^T\Sigma_{11}^{-1}({\bf x}_1-\mu_1)$$

$$A=\Sigma_{22}-\Sigma_{12}^T\Sigma_{11}^{-1}\Sigma_{12}$$