The Learning Law

Every time a particular pattern pair $(X_p, Y_p)$ is presented at the input layer, all the weights are modified in the following manner.

• Define error energy $E_p$ at the output layer:
  
  $$E_p=\frac{1}{2}\sum_{i=1}^m (y_{pi}-y'_{pi})^2=\frac{1}{2} \sum_{i=1}^m (\delta_{pi}^{o})^2$$
  
  
  where
  
  $$\delta_{pi}^{o} \stackrel{\triangle}{=}y_{pi}-y'_{pi}$$
  
  
  $y_{pi}$ is the desired output, and $y'_{pi}$ is the actual output
  
  $$y'_{pi}=f(net_{pi})=f(\sum_{j=1}^l w^{o}_{ij} z_{pj}+T_i)$$
  
  

• Find gradient of $E_p$ in output weight space $\{w_{ij}^{o}\}$:
  
  $$\frac{\partial E_p}{\partial w_{ij}^{o}} =\frac{\partial E... ...\partial w_{ij}^{o}} =-\delta_{pi}^{o}\;f'(net_{pi})\;z_{pj}$$
  
  

• Update $w_{ij}^{o}$ to minimize $E_p$ with gradient descent method:
  
  $$w_{ij}^{o(new)} =w_{ij}^{o(old)}+\Delta w_{ij}^{o} =w_{ij}^{o(old)} -\eta \frac{\partial E_p}{\partial w_{ij}^{o}}$$
  
  
  where
  
  $$\Delta w_{ij}^{o}\stackrel{\triangle}{=}-\eta \frac{\partial E_p}{\partial w_{ij}^{o}}$$
  
  
  and $\eta$ is the learning rate.
• Relate $E_p$ to the hidden layer weights $w_{jk}^{h}$
  

  $$E_p = \frac{1}{2}\sum_{i=1}^m(y_{pi}-y'_{pi})^2$$

  $$= \frac{1}{2}\sum_{i=1}^m(y_{pi}-f(\sum_{j=1}^lw_{ij}^{o}z_{pj}+T_i))^2$$

  $$= \frac{1}{2}\sum_{i=1}^m(y_{pi}-f(\sum_{j=1}^lw_{ij}^{o}f(net_{pj})+T_i))^2$$

  $$= \frac{1}{2}\sum_{i=1}^m(y_{pi}-f(\sum_{j=1}^lw_{ij}^{o}f(\sum_{k=1}^nw_{jk}^{h}x_{pk}+T_j)+T_i))^2$$

  
  

• Find gradient of $E_p$ in hidden weight space $\{w_{jk}^{h}\}$:
  

  $$\frac{\partial E_p}{\partial w_{jk}^{h}} = -\sum_{i=1}^m(y_{pi}-y'_{pi})\frac{\partial y'_{pi}}{\partial net... ...artial z_{pj}}{\partial net_{pj}} \frac{\partial net_{pj}}{\partial w_{jk}^{h}}$$

  $$= -\sum_{i=1}^m \delta_{pi}^{o} f'(net_{pi})w_{ij}^{o}\;f'(net_{pj})x_{pk}$$

  $$= -f'(net_{pj})x_{pk} \delta_{pj}^{h}$$

  
  
  where
  
  $$\delta_{pj}^{h} \stackrel{\triangle}{=}\sum_{i=1}^m \delta_{pi}^{o} f'(net_{pi})w_{ij}^{o}$$
  
  

• Update $w_{ij}^{h}$ to minimize $E_p$ with gradient descent method
  
  $$w_{jk}^{h(new)} =w_{jk}^{h(old)}+\Delta w_{jk}^{h} =w_{jk}^{h(old)} -\eta \frac{\partial E_p}{\partial w_{jk}}$$
  
  
  where
  
  $$\Delta w_{jk}^{h}\stackrel{\triangle}{=} -\eta \frac{\part... ...{\partial w_{ji}^{h}} =\eta f'(net_{pj}) x_k \delta_{pj}^{h}$$
  
  

Summary of Back Propagation Training

The following is for one training pattern pair $\{X_p, Y_p\}$.

1. Apply $X_p=(x_{p1}, x_{p2}, ... , x_{pn})^t$ to the input nodes;

2. Compute net input to hidden nodes
   
   $$net_{pj}=\sum_{k=1}^{n} w_{jk}^{h} x_{pk}+T_j$$
   
   

3. Compute output from hidden nodes
   
   $$z_{pj}=f(net_{pj})$$
   
   

4. Compute net input to output nodes
   
   $$net_{pi}=\sum_{j=1}^{l} w_{ij}^{o} z_{pj} + T_i$$
   
   

5. Compute output from output nodes
   
   $$y'_{pi}=f(net_{pi})$$
   
   

6. Find error terms for all output nodes (not quite the same as defined previously)
   
   $$\delta_{pi}^{o}=f'(net_{pi}) (y_{pi}-y'_{pi}) \;\;\;(i=1,...,m)$$
   
   
   where $Y_p=(y_{p1},y_{p2},...,y_{pm})^t$ is the desired output for $X_p$.
7. Find error terms for all hidden nodes (not quite the same as defined previously)
   
   $$\delta_{pj}^{h}=f'(net_{pj}) \sum_{i=1}^{m} \delta_{pi}^{o} w_{ij}^o \;\;\;(j=1,...,l)$$
   
   

8. Update weights to output nodes
   
   $$w_{ij}^o \leftarrow w_{ij}^o+\eta \delta_{pi}^o z_{pj}$$
   
   

9. Update weights to hidden nodes
   
   $$w_{jk}^h \leftarrow w_{jk}^h+\eta \delta_{pj}^h x_{pk}$$
   
   

10. Compute
    
    $$E_p=\frac{1}{2}\sum_{i=1}^{m} (y_{pi}-y'_{pi})^2$$
    
    

When this error is acceptably small for all of the training pattern pairs, training can be discontinued.

Competitive learning Network