Appendix B: Jensen's Inequality

$$\log(\sum_i a_i x_i) \ge \sum_i a_i \log(x_i)$$

The case of two terms in the summation can be shown graphically. An arbitrary point $x$ between $x_1$ and $x_2$ can be expressed as $x=a_1 x_1+a_2 x_2$ with the weighting coefficients $a_1$ and $a_2$ satisfying $a_1+a_2=1$. The weighted average $a_1 \log(x_1)+a_2 \log(x_2)$ of $\log(x_1)$ and $\log(x_2)$ is on the straight line between $\log(x_1)$ and $\log(x_2)$, below the point on the log function $\log(x)=\log(a_1 x_1+a_2 x_2)$, i.e.,

$$\log(x)=\log(a_1 x_1+a_2 x_2) \ge a_1 \log(x_1)+a_2 \log(x_2)$$

The equal sign only holds when either of the two weights is 1 and the other is 0. This result can be generalized to multiple terms in the weighted sum, the average of log functions of multiple $x_i$'s is always no greater than the log of the average of these $x_i$'s.