Error back propagation

The training process is composed of both forward pass (shown above) and the backward pass -- the error back propagation. The following is the 2-step training process for a particular pattern pair $({\bf x}_p, {\bf y}_p)$, which is repeated many times in random order for all $K$ pattern pairs.

• Define error energy $\varepsilon$:

  The actual output of the ith output node with pth pattern as the input is:
  

  $$y'_i=g(net^o_i)=g\left(\sum_{j=1}^L w^{o}_{ij} z_j+T^o_i\right),\;\;\;\;\;(i=1,\cdots,N)$$
  
  
  Let the desired output be $y_i$, the difference between actual and desired output is
  
  $$\delta_i^{o} \stackrel{\triangle}{=}y_i-y'_i$$
  
  
  The error energy is then defined as:
  
  $$\varepsilon=\frac{1}{2}\sum_{i=1}^M (y_i-y'_i)^2=\frac{1}{2} \sum_{i=1}^M(\delta_i^{o})^2$$
  
  
  Or in vector form:
  
  $$\varepsilon=\frac{1}{2}\vert\vert{\bf y}-{\bf y}'\vert\vert^2$$
  
  

• Find gradient of $\varepsilon$ in output weight space $w_{ij}^{o}$ ( $i=1,\cdots,M,\;j=1,\cdots,L$):
  
  $$\frac{\partial \varepsilon}{\partial w_{ij}^o} =\frac{\... ...t^o_i}{\partial w_{ij}^{o}} =-\delta_i^o\;g'(net^o_i)\;z_j$$
  
  
  Update $w_{ij}^{o}$ to minimize $\varepsilon$ with gradient descent method:
  
  $$w_{ij}^{o(new)}=w_{ij}^{o(old)}-\eta\;\frac{\partial e}{\pa... ...{ij}^o} =w_{ij}^{o(old)} +\eta\;\delta_i^o\;g'(net^o_i)\;z_j$$
  
  
  Here $\eta$ is the learning rate or step size. Or in matrix form:
  
  $${\bf W}^{o(new)}_{M\times L} ={\bf W}^{o(old)}_{M\times L}+\eta {\bf d}^o_{M\times 1}({\bf z}^T)_{1\times L}$$
  
  
  where ${\bf d}^o_{M\times 1}=[\delta^o_1g'(net^o_1),\cdots, \delta^o_Mg'(net^o_M)]^T$ is an $M\times 1$ vector whose elements are the pairwise products of ${\bf y}-{\bf y}'$ and ${\bf g}({\bf net}^o)$.

• Find gradient of $\varepsilon$ with respect to $T^o_i$ ($i=1,\cdots,M$):
  
  $$\frac{\partial \varepsilon}{\partial T_i^o} =\frac{\par... ...rtial net^o_i}{\partial T^o_i} =-\delta_i^o\;g'(net^o_i)$$
  
  
  Update $T^o_i$ to minimize $\varepsilon$ with gradient descent method:
  
  $$T^{o(new)}_i=T^{o(old)}_i+\Delta T_i^o =w_{ij}^{o(old)}-\e... ...artial T_i^o} =w_{ij}^{o(old)}+\eta\;\delta_i^o\;g'(net^o_i)$$
  
  
  Note that $T^o_i$ can be considered as a weight $T^o_i=w^o_{i0}$ connecting the ith output node to $z_0=1$.
• Relate $\varepsilon$ to the hidden layer weights $w_{jk}^h$ ( $j=1,\cdots,L,\;k=1,\cdots,N$):
  

  $$\varepsilon = \frac{1}{2}\sum_{i=1}^M(y_i-y'_i)^2 =\frac{1}{2}\sum_{i=1}^M(y_... ...2}\sum_{i=1}^M\left[y_i-g\left(\sum_{j=1}^L w_{ij}^{o}z_j+T^o_i\right)\right]^2$$

  $$= \frac{1}{2}\sum_{i=1}^M\left[y_i-g\left(\sum_{j=1}^L w_{ij}^o\,g(... ... w_{ij}^o\, g\left(\sum_{k=1}^N w_{jk}^hx_k+T^h_j\right)+T^o_i\right)\right]^2$$

  
  

• Find gradient of $\varepsilon$ in hidden weight space $w_{jk}^h$:
  

  $$\frac{\partial \varepsilon}{\partial w_{jk}^h} = \frac{\partial \varepsilon}{\partial y'_i}\; \frac{\partial ... ...al z_j}{\partial net^h_j}\; \frac{\partial net^h_j}{\partial w_{jk}^h}$$

  $$= -\sum_{i=1}^M \delta_i^o g'(net^o_i)w_{ij}^o\;g'(net^h_j)x_k =-\delta_j^{h}\;g'(net^h_j)x_k$$

  
  
  where
  
  $$\delta_j^h \stackrel{\triangle}{=}\sum_{i=1}^M \delta_i^o g'(net^o_i)w_{ij}^o$$
  
  
  Update $w_{ij}^{h}$ to minimize $\varepsilon$ with gradient descent method:
  
  $$w_{jk}^{h(new)}=w_{jk}^{h(old)} -\eta\;\frac{\partial e}{\p... ...l w_{jk}} =w_{jk}^{h(old)}+\eta\;\delta_j^h\;g'(net^h_j)x_k$$
  
  
  Or in matrix form:
  
  $${\bf W}^{h(new)}_{L\times N}={\bf W}^{h(old)}_{L\times N}+\eta {\bf d}^h_{L\times 1} {\bf x}^T_{1\times N}$$
  
  
  where ${\bf d}^h_{L\times 1}$ is an $L\times 1$ vector whose elements are the pairwise products of $({\bf W}^o)^T_{L\times M}{\bf d}^o_{M\times 1}$ and ${\bf g}'({\bf net}^o)$.

• Find gradient of $\varepsilon$ with respect to $T^h_j$ ($j=1,\cdots,l$):
  
  $$\frac{\partial \varepsilon}{\partial T_j^h}=-\delta_j^h\;g'(net^h_j)$$
  
  
  Update $T^h_j$ to minimize $\varepsilon$ with gradient descent method:
  
  $$T^{h(new)}_j=T^{h(old)}_j-\eta\;\frac{\partial \varepsilon}{\partial T_j^h} =T^{o(old)}_j+\eta\;\delta_j^h\;g'(net^h_j)$$
  
  
  Note that $T^h_j$ can be considered as a weight $T^h_j=w^h_{j0}$ connecting the jth hidden node to an input $x_0=1$.