Entropy

The entropy of a distribution $p_x(x)$ is defined as

$$H(X)=-\int p(x)\; log\; p(x) dx=E\{ -log\;p(X) \}$$

Entropy represents the uncertainty of the random variable. Among all distributions, uniform distribution has maximum entropy over a finite region $[a,b]$, while Gaussian distribution has maximum entropy over the entire real axis.

The joint entropy of two random variables $X$ and $Y$ is defined as

$$H(X,Y)=-\int p(x,y)\;log\;p(x,y) dx\;dy=E\{-log\;p(X,Y)\}$$

The conditional entropy of $X$ given $y$ is

$$H(X\vert y)=-\int p(x\vert y)\;log\;p(x\vert y) dx=E\{-log\;p(X\vert Y) \;\vert\; Y=y\}$$

and the conditional entropy of $X$ given $Y$ is

$$H(X\vert Y) = \int p(y) H(X\vert y) dy=-\int p(y) \int p(x\vert y)\;log\;p(x\vert y) dx dy$$

$$= -\int \int p(x,y)\;log\;p(x\vert y) dx dy =E\{E\{-log\;p(X\vert Y) \;\vert\; Y\} \}$$