Hebb's Learning

Donald Hebb (Canadian Psychologist) speculated in 1949 that

``When neuron A repeatedly and persistently takes part in exciting neuron B, the synaptic connection from A to B will be strengthened.''

Simultaneous activation of neurons leads to pronounced increases in synaptic strength between them. In other words, "Neurons that fire together, wire together. Neurons that fire out of sync, fail to link". So a Hebbian network can be used as an associator which will establish the association between two sets of patterns $\{{\bf x}_k,\;\;k=1,\cdots,K \}$ and $\{{\bf y}_k,\;\;k=1,\cdots,K\}$.

The classical conditioning (Pavlov, 1927) could be explained by Hebbian learning:

• F: sight of food - unconditioned stimulus
• B: sound of bell - conditioned stimulus
• S: Salivation - response

$$(F \rightarrow S) \Rightarrow (F \cap B \rightarrow S) \Rightarrow (B \rightarrow S)$$

Synaptic connections between pattern B and pattern S are strengthened as both are repeatedly excited simultaneously.

The structure

The Hebbian network model has an n-node input layer ${\bf x}=[x_1,\cdots,x_n]^T$ and an m-node output layer ${\bf y}=[y_1,\cdots,y_m]^T$. Each output node is connected to all input nodes:

$$y_i=\sum_{j=1}^n w_{ij} x_j\;\;\;\;\;(i=1,\cdots,m)$$

or in matrix form, we have

$${\bf y}_{m\times 1}={\bf W}_{m\times n}{\bf x}_{n\timex 1}$$

The learning law

$$w_{ij}^{new}=w_{ij}^{old}+\eta x_j y_i\;\;\;\;(i=1,\cdots,m,\;j=1,\cdots,n)$$

or in matrix form:

$${\bf W}^{new}={\bf W}^{old}+\eta {\bf y} {\bf x}^T$$

Here $\eta$ is the learning rate, a parameter controlling how fast the weights get modified. The reasoning for this learning law is that when both $x_j$ and $y_i$ are high (activated), the weight $w_{ij}$ (synaptic connectivity) between them is enhanced according to Hebbian learning.

Training

If we assume $w_{ij}=0$ initially, $\eta=1$ and a set of $K$ pairs of patterns $\{ ({\bf x}_k,{\bf y}_k),\;\;k=1,\cdots,K \}$ are presented repeatedly during training, we have

$$w_{ij}=\sum_{k=1}^Kx_j^{(k)}y_i^{(k)}\;\;\;\;(i=1,\cdots,m,\;\;j=1,\cdots,n)$$

or in matrix form, the weight matrix is the sum of the outer-products of all $K$ pairs of patterns:

$${\bf W}_{m\timex n}=\sum_{k=1}^K {\bf y}_k {\bf x}_k^T = \s... ...m^{(k)} \end{array} \right] [ x_1^{(k)}, \cdots, x_n^{(k)} ]$$

Classification

When presented with one of the patterns ${\bf x}_l$, the network will produce the output:

$${\bf y}={\bf W}{\bf x}_l=(\sum_{k=1}^K {\bf y}_k {\bf x}_k^T... ...x}_l^T{\bf x}_l)+\sum_{k\neq l}{\bf y}_k({\bf x}_k^T{\bf x}_l)$$

To interpret the output pattern ${\bf y}$, we first consider the ideal case where the following two conditions are satisfied:

• ${\bf x}$'s are orthogonal to each other:
  
  $${\bf x}_i^T{\bf x}_j=\vert{\bf x}_i\vert \vert{\bf x}_j\vert ... ...{ \begin{array}{ll} 1 & i=j 0 & i\ne j \end{array} \right.$$
  
  
  where $\vert{\bf x}_i\vert$ is the length of vector ${\bf x}_i$ and $\phi$ is the angle between vectors ${\bf x}_i$ and ${\bf x}_j$.

  This condition implies that the $K$ input patterns are totally unrelated to each other. Intuitively, how much two vectors are related to each other can be measured by the angle $\phi$ between them. If $\phi$ is close to 0 the two vectors are closely related, because their elements are very similar to each other so that they almost coincide. When negative values are allowed for the elements and $\phi$ is close to 180 degrees, the vectors are also highly related because their elements are the opposite of each other. In either case, the $cos \phi$ is close to 1 and the inner product is maximized indicating that the two vectors are either positively or negatively related to each other. On the other hand, if the two vectors are perpendicular to each other ($\phi=\pi/2$, $cos \phi = 0$), the inner product of the two vectors is very small or even zero indicating the elements of the a vector are irrelevant to those of the other vector.

  How much two patterns ${\bf x}$ and ${\bf y}$ are related to each other can also be measured quantitatively, we define correlation coefficient as:
  

  $$r_{xy}\stackrel{\triangle}{=}\frac{{\bf x}^T{\bf y}}{\vert\... ...rac{\sum_i x_i y_i}{\sqrt{\sum_i x_i^2}\sqrt{\sum_i y_i^2} }$$
  
  
  $r_{xy}$ is the inner product of two normalized vectors representing two patterns in the n-D feature space, and can take the following values

  $0 < r_{xy} \leq 1$	if ${\bf x}$ and ${\bf y}$ are positively correlated ($r_{xy}=1$ iff ${\bf x}={\bf y}$)
  $r_{xy} =0$	if ${\bf x}$ and ${\bf y}$ are not correlated
  $-1<r_{xy} \leq 0$	if ${\bf x}$ and ${\bf y}$ are negatively correlated ($r_{xy}=-1$ iff ${\bf x}=-{\bf y}$)

  In other words, if ${\bf x}$ and ${\bf y}$ are totally unrelated, then $r_{xy} =0$ and they are orthogonal to each other.

• The number of pattern pairs is smaller than the number of input nodes:
  
  $$K \leq n$$
  
  

  This condition implies that the capacity of the network ($n$ input nodes) is large enough for representing $K$ different patterns. This can be translated into mathematical language that there can be no more than $n$ orthogonal vectors in an n-D space.

If these conditions are true, then the response of the network to ${\bf x}_l$ will be

$${\bf y}={\bf y}_l({\bf x}_l^T{\bf x}_l)+\sum_{k\neq l}{\bf y}_k({\bf x}_k^T{\bf x}_l)={\bf y}_l$$

because all other terms ($k \ne l$) are zero:

$${\bf x}_k^T {\bf x}_l=0\;\;\;\;for\;\;all\;\;k \neq l$$

In other words, a one-to-one correspondence relationship between ${\bf x}_k$ and ${\bf y}_k$ has been established for all $k=1,\cdots,K$. In non-ideal case, the summation term (called cross talk) is non-zero and we have an error ${\bf y}-{\bf y}_l \neq 0$.

Although two patterns ${\bf x}=B$ and ${\bf y}=S$ don't have causal relationship (as $S$ is caused by another pattern $F$), but if they always appear simultaneously, an association relationship can be established in the Hebbian network.