Error back propagation

The training process is composed of both forward pass (shown above) and the backward pass — the error back propagation. The following is the 2-step training process for a particular pattern pair $({\bf x}_p, {\bf y}_p)$, which is repeated many times in random order for all $K$ pattern pairs.

• Define error energy $\varepsilon$:

  The actual output of the ith output node corresponding to the pth input pattern is:

  $$y'_i=g(net^o_i)=g\left(\sum_{j=1}^L w^{o}_{ij} z_j+T^o_i\right), \;\;\;\;\;(i=1,\cdots,M)$$
  Given the desired output $y_i$, we find the difference between actual and desired output:
  $$\delta_i^{o} \stackrel{\triangle}{=}y_i-y'_i$$
  and define the error function to be:
  $$\varepsilon=\frac{1}{2} \sum_{i=1}^M(\delta_i^{o})^2 =\frac{1}{2}\sum_{i=1}^M (y_i-y'_i)^2 =\frac{1}{2}\vert\vert{\bf y}-{\bf y}'\vert\vert^2$$

• Find gradient of the error $\varepsilon$ with respect to $T^o_i$ ($i=1,\cdots,M$) by the chain rule:
  $$\frac{\partial\,\varepsilon}{\partial\, T_i^o} =\frac{\partial\,... ..._i}\; \frac{\partial\, net^o_i}{\partial\, T^o_i} =-\delta_i^o\;g'(net^o_i)$$
  Update $T^o_i$ to minimize $\varepsilon$ with gradient descent method:
  $$T^{o(new)}_i=T^{o(old)}_i+\Delta T_i^o =w_{ij}^{o(old)}-\eta\;\frac{\partial e}{\partial T_i^o} =w_{ij}^{o(old)}+\eta\;\delta_i^o\;g'(net^o_i)$$
  Note that $T^o_i$ can be considered as a weight $T^o_i=w^o_{i0}$ connecting the ith output node to $z_0=1$.
• Relate $\varepsilon$ to the hidden layer weights $w_{jk}^h$ ( $j=1,\cdots,L,\;k=1,\cdots,N$):
  

  $$\varepsilon = \frac{1}{2}\sum_{i=1}^M(y_i-y'_i)^2 =\frac{1}{2}\sum_{i=1}^M(y_i-... ...2}\sum_{i=1}^M\left[y_i-g\left(\sum_{j=1}^L w_{ij}^{o}z_j+T^o_i\right)\right]^2$$

  $$= \frac{1}{2}\sum_{i=1}^M\left[y_i-g\left(\sum_{j=1}^L w_{ij}^o\,g(... ...L w_{ij}^o\, g\left(\sum_{k=1}^N w_{jk}^hx_k+T^h_j\right)+T^o_i\right)\right]^2$$

• Find gradient of $\varepsilon$ in hidden weight space $w_{jk}^h$:
  

  $$\frac{\partial \varepsilon}{\partial w_{jk}^h} = \frac{\partial\,\varepsilon}{\partial\, y'_i}\; \frac{\partial\, ... ...ial\, z_j}{\partial\, net^h_j}\; \frac{\partial\, net^h_j}{\partial\, w_{jk}^h}$$

  $$= -\sum_{i=1}^M \delta_i^o g'(net^o_i)w_{ij}^o\;g'(net^h_j)x_k =-\delta_j^{h}\;g'(net^h_j)x_k$$

  where
  $$\delta_j^h \stackrel{\triangle}{=}\sum_{i=1}^M \delta_i^o g'(net^o_i)w_{ij}^o$$
  Update $w_{ij}^{h}$ to minimize $\varepsilon$ with gradient descent method:
  $$w_{jk}^{h(new)}=w_{jk}^{h(old)} -\eta\;\frac{\partial e}{\partial w_{jk}} =w_{jk}^{h(old)}+\eta\;\delta_j^h\;g'(net^h_j)x_k$$
  Or in matrix form:
  $${\bf W}^{h(new)}_{L\times N}={\bf W}^{h(old)}_{L\times N}+\eta {\bf d}^h_{L\times 1} {\bf x}^T_{1\times N}$$
  where ${\bf d}^h_{L\times 1}$ is an $L\times 1$ vector whose elements are the pairwise products of $({\bf W}^o)^T_{L\times M}{\bf d}^o_{M\times 1}$ and ${\bf g}'({\bf net}^o)$.

• Find gradient of $\varepsilon$ with respect to $T^h_j$ ($j=1,\cdots,l$):
  $$\frac{\partial\,\varepsilon}{\partial\, T_j^h}=-\delta_j^h\;g'(net^h_j)$$
  Update $T^h_j$ to minimize $\varepsilon$ with gradient descent method:
  $$T^{h(new)}_j=T^{h(old)}_j-\eta\;\frac{\partial\,\varepsilon}{\partial\,T_j^h} =T^{o(old)}_j+\eta\;\delta_j^h\;g'(net^h_j)$$
  Note that $T^h_j$ can be considered as a weight $T^h_j=w^h_{j0}$ connecting the jth hidden node to an input $x_0=1$.

Applications of BP networks include:

• Pattern recognition/classification
• Medical diagnosis (symptoms, diagnosis, drugs)
• Time series analysis (curve-fitting, interpolation and extrapolation, e.g., stock-market prediction).
• Data compression
• Automation (robots, autonomous vehicles)
• .....

A story related (maybe) to neural networks

An elementary school teacher asked her class to give examples of the great technological inventions in the 20th century. One kid mentioned it was the telephone, as it would let you talk to someone far away. Another kid said it was the airplane, because an airplane could take you to anywhere in the world. Then the teacher saw little Johnny eagerly waving his hand in the back of the classroom. “What do you think is the greatest invention, Johnny?” “The thermos!” The teacher was very puzzled, “Why do you think the thermos? All it can do is to keep hot things hot and cold things cold.” “But”, Johnny answered, “ how does it know when to keep things hot and when to keep them cold?”

Don't you sometimes wonder “How does a neural network know ...?”

Well, it does not. It is just a monkey-see-monkey-do type of learning, like this...