Nonlinar Regression

To enhance the expressive power of the linear regression model $f({\bf x})={\bf w}^T{\bf x}$, it can be generalized to a nonlinear model based on a set of $H$ basis functions $\phi_h({\bf x})$ ($h=1,\cdots,H$):

$$y({\bf x},{\bf w})=\sum_{h=1}^H w_h \phi_h({\bf x}) ={\bf w}^T {\bf\phi}({\bf x})$$

where

$${\bf\phi}({\bf x})=[\phi_1({\bf x}),\cdots,\phi_H({\bf x})]^T$$

The dimensionality of the weight vector ${\bf w}$ has changed from $d$ (dimensionality of ${\bf x}$) to $H$ (number of basis functions).

For example, when the dimensionality of ${\bf x}$ is $d=1$,the basis function can be

$$\phi_h(x)=exp[-\frac{(x-c_h)^2}{2r^2}]\;\;\;\;(h=1,\cdots,H)$$

each centered around some point $c_h$. Alternatively, the basis functions of a scalar $x$ can be

$${\bf\phi}(x)=(1,x,x^2,\cdots)^T$$

In such cases, the previous derivation is still valid if we replace ${\bf x}$ by ${\bf\phi}({\bf x})$ and the predictive distribution is

$$p(y^*\vert{\bf x}^*,{\bf X},{\bf y})= N(\mu_w^T {\bf\phi}({\bf x}^*), {\bf\phi}({\bf x})^{*T}\Sigma_w{\bf\phi}({\bf x}^*))$$

where

$$\mu_w=\frac{1}{\sigma_n^2}\Sigma_w^{-1} {\bf\Phi}{\bf y}$$

where

$${\bf\Phi}({\bf X})=[ {\bf\phi}_1({\bf X}),\cdots,{\bf\phi}_H(... ... ... & \phi_H({\bf x}^{(N)}) \end{array} \right]_{N\times H}$$

with each column for each of the $H$ basis functions ${\bf\phi}_h({\bf x})$ containing the $N$ function values at the $N$ input data points ${\bf x}^{(n)}$ ($i=1,\cdots,N$):