Bayesian Inference

The goal of Bayesian inference is to find the model parameters denoted by $\theta$, based on the observed data denoted by $D$. Assume the a priori distribution of the parameters is $P(\theta)$ and the distribution of the data is $P(D)$, then the joint probability of both the data and parameters is

$$P(D,\theta)=P(D\vert\theta) P(\theta)$$

The posterior distribution of the model parameters can be obtained according to Bayesian theorem:

$$P(\theta\vert D)=\frac{P(\theta)P(D\vert\theta)}{P(D)} =\fra... ...theta)P(D\vert\theta)}{\int P(\theta) P(D\vert\theta) d\theta}$$

where $P(D\vert\theta)=L(\theta\vert D)$ is the likelihood function of the parameters $\theta$, given the observed data $D$. This equation can be interpreted as

$$\mbox{posterior}=\frac{\mbox{likelihood} \times \mbox{prior}}... ...zation factor}} \propto \mbox{prior} \times \mbox{likelihood}$$

In a maximum-likelihood problem, the goal is to find $\theta$ that maximizes the likelihood $L$:

$$\theta^*=argmax_{\theta}\;L(\theta\vert D)$$

which can be obtained by solving the likelihood equation:

$$\frac{\partial}{\partial \theta} L(\theta\vert D) =\frac{\partial}{\partial \theta} p(D\vert\theta)=0$$

Bayesian inference can be used to find any feature of the posterior distribution $f(\theta)$, whose posterior expectation is

$$E[f(\theta)\vert D]=\int f(\theta) P(\theta\vert D) d\theta =... ...(D\vert\theta)d\theta}{\int P(\theta) P(D\vert\theta) d\theta}$$

The integration in this expression is likely to be of high dimensions, and in most applications, analytical evaluation of $E[f(\theta)\vert D]$ is impossible. In such cases, Monte Carlo integration can be used, including Markov Chain Monte Carlo (MCMC).