Biological and artificial networks

The Brain:

consists of $10^{11}$ nerve cells (also called neurons) interconnected through about $10^{15}$ synaptic junctions to form millions of neural networks. Hundreds specialized cortical areas are formed based on these networks for different information processing tasks;

The Neuron:

• Dendrites:

  conduct impulses (inputs) from other cells toward the cell body;

• Axon:

  conducts impulses (output) away from the cell body to other cells;

• Synapse:

  the point at which impulses pass from one cell to another;

See additional notes

Model of neuron:

A neuron is modeled mathematically as the following:

• Activation - the net input signal:

  The net input to the ith node is a weighted sum of all inputs:
  

  $$net_i=\sum_j w_{ij} x_j$$
  
  
  where $o_j$ is the input signal from the jth node, $w_{ij}$ is the synaptic connectivity between the jth node and the ith node.

  
  

  $$w_{ij}=\left\{ \begin{array}{ll} > 0 & excitatory\;\;input \... ...ory\;\;input \\ = 0 & no\;\;connection \end{array} \right.$$
  
  

• Output signal:

  The output of the ith node is a function of the net input:
  

  $$y_i=g(net_i)=g\left(\sum_j w_{ij} x_j\right)$$
  
  
  where $g(x)$ is an activation function which can be a sigmoid function, as plotted in the figure below. This function can be either one-sided $0\le g(x)\le 1$ (top) or two-sided $-1\le g(x)\le 1$ (bottom):
  • One-sided:
    
    $$g(x,a)=\frac{1}{1+exp(-ax)},\;\;\;\;\;\;g'(x)=\frac{a exp(-ax)}{(1+exp(-ax))^2}$$
    
    

  • Two-sided:
    
    $$g(x,a)=\frac{2}{1+exp(-ax)}-1,\;\;\;\;\;\;g'(x)=\frac{2a exp(-ax)}{(1+exp(-ax))^2}$$
    
    

  where $a$ is a parameter that controls the slop of $g(x)$. Specially, when $a\rightarrow 0$, $g(x,a)$ becomes linear, but whn $a\rightarrow \infty$, $g(x,a)$ becomes a threshold function:
  
  $$\lim_{a\rightarrow\infty} g(x,a)=\left\{ \begin{array}{cl} 0\mbox{ or }-1 & x<0\\ 1 & x>0 \end{array} \right.$$
  
  

  

Model of neural network:

The artificial neural network is a group of neurons organized in several layers:

• Input layer: receives inputs from sources external to the network;
• Output layer: generates outputs to the external world.
• Hidden layer(s): layers in between of the input and output layers, not visible from outside the network.
• Learning laws: mathematical rules for modifying the weights of a network iteratively according the inputs (and outputs if the learning is supervised).

Computation

Assume a simple 2-layer network has $n$ input nodes receiving stimuli (input patterns) ${\bf x}=[x_1,\cdots,x_n]^T$ and $m$ output nodes generating responses (output patterns) ${\bf y}=[y_1,\cdots,y_m]^T$. Also assume the output function $f$ is an identical function. Then the output of the ith output node is:

$$y_i=\sum_{j=1}^n w_{ij} x_j={\bf w}_i^T {\bf x} \;\;\;\;\;(i=1,\cdots,m)$$

where

$${\bf w}_i\stackrel{\triangle}{=}[w_{i1},\cdots,w_{in} ]^T$$

is a weight vector formed by all the weights leading from the $n$ input nodes to the ith output node.

The output ${\bf y}$ can also be represented in the matrix form:

$${\bf y}={\bf W}{\bf x}$$

where ${\bf W}$ is the weight matrix

$${\bf W}=\left[ \begin{array}{c} {\bf w}_1^T \vdots {\b... ...w_{m2} & \cdots & w_{mn} \\ \end{array} \right]_{m \times n}$$