Linear separation of a feature space

A hyper plane in an n-D feature space can be represented by the following equation:

$$f({\bf x})={\bf x}^T {\bf w}+b=\sum_{i=1}^n x_i w_i+b=0$$

Dividing by $\vert\vert{\bf w}\vert\vert=\sqrt{{\bf w}^T{\bf w}}$, we get

$$\frac{{\bf x}^T {\bf w}}{\vert\vert{\bf w}\vert\vert}=P_{\bf... ...vert\vert{\bf w}\vert\vert}=-\frac{b}{\sqrt{{\bf w}^T{\bf w}}}$$

indicating that the projection of any point ${\bf x}$ on the plane onto the vector ${\bf w}$ is always $-b/\vert\vert{\bf w}\vert\vert$, i.e., ${\bf w}$ is the normal direction of the plane, and $\vert b\vert/\vert\vert{\bf w}\vert\vert$ is the distance from the origin to the plane. Note that the equation of the hyper plane is not unique. $c f({\bf x})=0$ represents the same plane for any $c$.

The n-D space is partitioned into two regions by the plane. Specifically, we define a mapping function $y=sign(f({\bf x})) \in \{1,-1\}$,

$$f({\bf x})={\bf x}^T {\bf w}+b=\left\{ \begin{array}{ll} >0,... ... y=sign(f({\bf x}))=-1,\;{\bf x}\in N \\ \end{array} \right.$$

Any point ${\bf x}\in P$ on the positive side of the plane is mapped to 1, while any point ${\bf x}\in N$ on the negative side is mapped to -1. A point ${\bf x}$ of unknown class will be classified to P if $f({\bf x})>0$, or N if $f({\bf x})<0$.

Example:

A straight line in 2D space ${\bf x}=[x_1, x_2]^T$ described by the following equation:

$$f({\bf x})={\bf x}^T {\bf w}+b=[x_1,x_2] \left[ \begin{array... ...t[ \begin{array}{c} 1 2 \end{array} \right]-1 =x_1+2x_2-1=0$$

devides the 2D plane into two halves. The distance between the origin and the line is

$$\frac{\vert b\vert}{\vert\vert{\bf w}\vert\vert}=\frac{1}{\sqrt{w_1^2+w_2^2}}=\frac{1}{\sqrt{5}}=0.447$$

Consider three points:

• ${\bf x}_0=[0.5,\;0.25]^T$, $f({\bf x}_0)=0.5+2\times 0.25-1=0$, i.e., ${\bf x}_0$ is on the plane;
• ${\bf x}_1=[1,\;0.25]^T$, $f({\bf x}_1)=1+2\times 0.25-1=0.5>0$, i.e., ${\bf x}_1$ is above the straight line;
• ${\bf x}_2=[0.5,\;0]^T$, $f({\bf x}_2)=0.5+2\times 0-1=-0.5<0$, i.e., ${\bf x}_2$ is below the straight line.