Covariance and Correlation

The mean and the variance of two variables $x$ and $y$ can be estimated by averaging the outcomes of the random experiment concerning the variables repeated $K$ times:

$$\hat{\mu}_x=\frac{1}{K}\sum_{i=1}^K x_i,\;\;\;\; \hat{\sigma}^2_{x}=\frac{1}{K}\sum_{i=1}^K (x_i-\hat{\mu}_x)^2$$

$$\hat{\mu}_y=\frac{1}{K}\sum_{i=1}^K y_i,\;\;\;\; \hat{\sigma}^2_{y}=\frac{1}{K}\sum_{i=1}^K (y_i-\hat{\mu}_y)^2$$

and their covariance can be estimated as

$$\hat{\sigma}^2_{xy}=\frac{1}{K}\sum_{i=1}^K (x_i-\hat{\mu}_x)(y_i-\hat{\mu}_y)$$

Examples

Consider the following examples showing the covariance $\sigma_{xy}$ between two random variables $x$ and $y$ under various situations.

• Example 1: assume the experiment concerning $x$ and $y$ is repeated $K=3$ times with the outcomes:
  
  $$\begin{tabular}{l\vert lll}\hline & 1st & 2nd & 3rd \hline x & 1 & 2 & 3 \\ y & 1 & 2 & 3 \hline \end{tabular}$$
  
  
  The means and covariance of $x$ and $y$ can be estimated as
  
  $$\hat{\mu}_x=\frac{1}{K}\sum_{i=1}^Kx_i =\hat{\mu}_y=\frac{1}{K}\sum_{i=1}^Ky_i=2$$
  
  
  
  
  $$\hat{\sigma}^2_{xy}=\frac{1}{K}\sum_{i=1}^K (x_i-\hat{\mu}_x)... ...mu}_y) =(1 \times 1+2 \times 2+3 \times 3)/3-2 \times 2=0.667$$
  
  
  We see that in this case $x$ and $y$ are highly (maximally) correlated.

• Example 2: assume the outcomes of the 3 experiments are
  
  $$\begin{tabular}{l\vert lll}\hline & 1st & 2nd & 3rd \hline x & 1 & 2 & 3 \\ y & 3 & 2 & 1 \hline \end{tabular}$$
  
  
  Similarly, we get
  
  $$\hat{\mu}_x=\frac{1}{K}\sum_{i=1}^Kx_i =\hat{\mu}_y=\frac{1}{K}\sum_{i=1}^Ky_i=2$$
  
  
  
  
  $$\hat{\sigma}^2_{xy}=\frac{1}{K}\sum_{i=1}^K (x_i-\mu_x)(y_i-\mu_y) =(1 \times 3+2 \times 2+3 \times 1)/3-2 \times 2=-0.667$$
  
  
  indicating that the two variables are highly negatively correlated.

• Example 3: assume the outcomes are:
  
  $$\begin{tabular}{l\vert lll}\hline & 1st & 2nd & 3rd \hline x & 1 & 2 & 3 \\ y & 2 & 2 & 2 \hline \end{tabular}$$
  
  
  We have
  
  $$\hat{\mu}_x=\hat{\mu}_y=2$$
  
  
  
  
  $$\hat{\sigma}^2_{xy}=(1 \times 2+2 \times 2+3 \times 2)/3-2 \times 2=0$$
  
  
  indicating that the two variables are totally uncorrelated (unrelated).

• Example 4: assume the outcomes are:
  
  $$\begin{tabular}{l\vert lll}\hline & 1st & 2nd & 3rd \hline x & 2 & 2 & 2 \\ y & 1 & 2 & 3 \hline \end{tabular}$$
  
  
  We have
  
  $$\hat{\mu}_x=\hat{\mu}_y=2$$
  
  
  
  
  $$\hat{\sigma}^2_{xy}=(2 \times 1+2 \times 2+2 \times 3)/3-2 \times 2=0$$
  
  
  indicating that the two variables are totally uncorrelated (unrelated).

• Example 5: assume the experiment is carried out $K=5$ times (combination of examples 4 and 5) with the outcomes:
  
  $$\begin{tabular}{l\vert lllll}\hline & 1st & 2nd & 3rd & 4th... ...2 & 2 & 2 & 3 \\ y & 2 & 1 & 2 & 3 & 2 \hline \end{tabular}$$
  
  
  We still have
  
  $$\hat{\mu}_x=\hat{\mu}_y=2$$
  
  
  
  
  $$\hat{\sigma}^2_{xy}=(1 \times 2+2 \times 1+2 \times 2+2 \times 3+3 \times 2)/5 -2 \times 2=0$$
  
  
  indicating that the two variables are totally uncorrelated (unrelated).