Mutual information

Mutual information is defined as

$$I(X,Y) = H(X)+H(Y)-H(X,Y)$$

$$= E\{ -log\;\;p(X)\}+E\{ -log\;\;p(Y)\}+E\{ -log\;\;p(X,Y)\}$$

$$= E\{ log\;\frac{p(X,Y)}{p(X)\;p(Y)} \}$$

Mutual information measures the amount of information shared between the two random variables $X$ and $Y$. Since

$$p(X,Y)=p(X\vert Y) \; p(Y)=p(Y\vert X) \; p(X)$$

we have

$$I(X,Y) = E\{ log\;\frac{p(X,Y)}{p(X)\;p(Y)} \}$$

$$= E\{ log\;\frac{p(X\vert Y)}{p(X)} \}=H(X)-H(X\vert Y)$$

$$= E\{ log\;\frac{p(Y\vert X)}{p(Y)} \}=H(Y)-H(Y\vert X)$$