Entropy, uncertainty of random experiment

A random experiment may have binary (e.g., rain or dry) or multiple outcomes. For example, a dice has six possible outcomes with equal probability, or a pixel in a digital image takes one of the $2^8=256$ gray levels (0  255) with not necessarily the same probability. In general, these multiple outcomes can be considered as $N$ events $E_i$ with corresponding probability $P_i=P(E_i)$ ($i=1,\cdots,N$), which are

• complementary
  
  $$E_1 \bigcup E_2 \bigcup \cdots \bigcup E_n=\bigcup_{i=1}^N E_i = I$$
  
  
  where $E_i\bigcup E_j$ is the union (OR) of events $E_i$ and $E_j$, and $I$ is a necessary event with $P(I)=1$, so that
  
  $$P(\bigcup_{i=1}^N E_i)=\sum_{i=1}^N P(E_i)=P(I)=1$$
  
  

• mutually exclusive
  
  $$E_i \bigcap E_j = \Phi$$
  
  
  where $E_i\bigcap E_j$ is the intersection (AND) of events $E_i$ and $E_j$, and $\Phi$ is an impossible event with $P(\Phi)=0$, so that
  
  $$P(E_i,E_j)=P(\Phi)=0$$
  
  

The uncertainty about the outcome of such a random experiment is the sum of the uncertainty $H(E_i)$ associated with each individual event $E_i$, weighted by its probability $P_i$:

$$H(E_1, \cdots, E_N)\stackrel{\triangle}{=}\sum_{i=1}^N P_i\; H(E_i) =-\sum_{i=1}^N P_i \;log\; P_i$$

This is called the entropy which measures the uncertainty of the random experiment. Once the result of the experiment is known, the uncertainty becomes zero, i.e., the entropy is also the information associated with the experiment. The specific logarithmic base is not essential. If the base is 2, the unit of entropy is the bit; if the base is $e=2.71828$, the unit is nat (or nit).

For example, the weather can have two complementary and mutually exclusive possible outcomes: rain $E_1$ with probability $P_1$ or dry $E_2$ with probability $P_2=1-P_1$. The uncertainty of the weather is therefore the sum of the uncertainty of a rainy weather and the uncertainty of a fine weather weighted by their probabilities:

$$H(E_1, E_2) = P_1\; H(E_1)+P_2\; H(E_2)=-P_1 \;log\; P_1-P_2 \;log\; P_2$$

$$= -P_1\; log \;P_1-(1-P_1)\; log \;(1-P_1)$$

In particular, consider cases:

• If $P_1=1$ and $P_2=0$ (or vice versa), we have the minimum uncertainty zero (note that $x\; log \;x \vert _{x=0}=0$):
  
  $$H(E_1, E_2) = -1\; log_2 \;1-0\; log_2 \;0=0$$
  
  

• If $P_1=0.1$, $P_2=0.9$, the uncertainty is:
  
  $$H(E_1, E_2) = -(0.1\; log_2 \;0.1+0.9\; log_2 \;0.9)=0.47$$
  
  

• If $P_1=0.2$, $P_2=0.8$, the uncertainty is:
  
  $$H(E_1, E_2) = -(0.2\; log_2 \;0.2+0.8\; log_2 \;0.8)=0.72$$
  
  

• If $P_1=0.3$, $P_2=0.7$, the uncertainty is:
  
  $$H(E_1, E_2) = -(0.3\; log_2 \;0.3+0.7\; log_2 \;0.7)=0.88$$
  
  

• If $P_1=0.4$, $P_2=0.6$, the uncertainty is:
  
  $$H(E_1, E_2) = -(0.4\; log_2 \;0.4+0.6\; log_2 \;0.6)=0.97$$
  
  

• If $P_1=P_2=0.5=1/2$, we have the maximum uncertainty:
  
  $$H(E_1, E_2) = -\frac{1}{2}\; log_2 \;\frac{1}{2}-\frac{1}{2}\; log_2 \;\frac{1}{2}=\frac{1}{2}+\frac{1}{2}=1$$