Biological and artificial networks

The Brain:

consists of $10^{11}$ nerve cells (also called neurons) interconnected through about $10^{15}$ synaptic junctions to form millions of neural networks. Hundreds specialized cortical areas are formed based on these networks for different information processing tasks;

The Neuron:

• Dendrites:

  conduct impulses (inputs) from other cells toward the cell body;

• Axon:

  conducts impulses (output) away from the cell body to other cells;

• Synapse:

  the point at which impulses pass from one cell to another;

See additional notes

Model of neuron:

A neuron is modeled mathematically as the following:

• Activation – the net input signal:

  The net input or the activation received by the ith node is a weighted sum of all inputs:

  $$p=\sum_{i=1}^N w_i x_i+T={\bf w}^T{\bf x}+T$$
  where ${\bf x}=[x_1,\cdots,x_N]^T$ is the input vector representing the signal received by the node, and ${\bf w}=[w_1,\cdots,w_n]^T$ is the weight vector of the node representing its synaptic connectivity to each of the $N$ inputs:
  $$w_i\;\;\left\{ \begin{array}{ll} > 0 & excitatory\;\;input \\ < 0 & inhibitory\;\;input \\ = 0 & no\;\;connection \end{array} \right.$$
  and $T$ is a threshold of the node. When the net input received by the node exceeds $T$, it will generate strong response.

• Output signal:

  The output of each node is a activation function of the net input it receives:

  $$y=g(p)=g({\bf w}^T{\bf x}+T)=\frac{1}{1+e^{-({\bf w}^T{\bf x}+T)}}$$
  where $g(x)=1/(1+e^{-x})$ is the sigmoid function and its derivative
  $$g'(x)=\frac{ae^{-x}}{(1+e^{-x})^2} =\frac{1}{1+e^{-x}}\;\frac{1+e^{-x}-1}{1+e^{-x}}=g(x)\,(1-g(x))$$
  As a special 1-D case, $g(wx+T)$ and its derivative are plotted below. We see that the threshold $T$ shifts the sigmoid function horizontally, while the inner product ${\bf x}^T{\bf w}$ determines the slope of the sigmoid function.

  

  For computational convenience, the threshold term can be treated as the 0th term of the summation with $w_0=T$ and $x_0=1$:

  $$p=\sum_{i=1}^N w_i x_i+T=\sum_{j=0}^N w_i x_i ={\bf w}^T{\bf x}$$
  where we have redefined ${\bf x}=[x_0=1, x_1,\cdots,x_N]^T$ and ${\bf w}=[w_0=T, w_1,\cdots,w_N]^T$.

Model of neural network:

The artificial neural network is a group of neurons organized in several layers:

• Input layer: receives inputs from sources external to the network;
• Output layer: generates outputs to the external world.
• Hidden layer(s): layers in between of the input and output layers, not visible from outside the network.
• Learning laws: mathematical rules for modifying the weights of a network iteratively according the inputs (and outputs if the learning is supervised).

Computation

Assume a simple 2-layer network has $n$ input nodes receiving stimuli (input patterns) ${\bf x}=[x_1,\cdots,x_n]^T$ and $m$ output nodes generating responses (output patterns) ${\bf y}=[y_1,\cdots,y_m]^T$. Also assume the output function $f$ is an identical function. Then the output of the ith output node is:

$$y_i=\sum_{j=1}^n w_{ij} x_j={\bf w}_i^T {\bf x} \;\;\;\;\;(i=1,\cdots,m)$$

where

$${\bf w}_i\stackrel{\triangle}{=}[w_{i1},\cdots,w_{in} ]^T$$

is a weight vector formed by all the weights leading from the $n$ input nodes to the ith output node.

The output ${\bf y}$ can also be represented in the matrix form:

$${\bf y}={\bf W}{\bf x}$$

where ${\bf W}$ is the weight matrix

$${\bf W}=\left[ \begin{array}{c} {\bf w}_1^T \\ \vdots \\ {\bf w}... ...s \\ w_{m1} & w_{m2} & \cdots & w_{mn} \\ \end{array} \right]_{m \times n}$$