Some special cases

If $p({\bf x}/\omega_i)$ can be assumed to be a normal distribution, the discriminant function can be written as:

$$D_i({\bf x})=P(\omega_i)p({\bf x}/\omega_i) =P(\omega_i)\; \... ... x}-{\bf m}_i)^T {\bf\Sigma}_i^{-1}({\bf x}-{\bf m}_i)\right]$$

Since $D_i({\bf x})$ is used only relatively among all classes, it can be replaced by a monotonic log function and the discriminant function becomes

$$\log\;D_i({\bf x}) = \log[ p({\bf x}/\omega_i) P(\omega_i)] = \log\; p({\bf x}/\omega_i) + log P(\omega_i)$$

$$= -\frac{1}{2}({\bf x}-{\bf m}_i)^T{\bf\Sigma}_i^{-1}({\bf x}-{\bf ... ...{2}\;log 2\pi -\frac{1}{2}\;log \vert{\bf\Sigma}_i \vert + log P(\omega_i)$$

The second term $-N log 2\pi/2$ is a constant common to all classes and can therefore be dropped and the discriminant function is

$$D_i({\bf x})=-\frac{1}{2}({\bf x}-{\bf m}_i)^T\Sigma_i^{-1}(... ...frac{1}{2}\;log \vert{\bf\Sigma}_i \vert + log P(\omega_i)$$

This is the most general case and the boundary between any two classes $\omega_i$ and $\omega_j$ is described by $D_i({\bf x})=D_j({\bf x})$, i.e.,

$$-\frac{1}{2}({\bf x}-{\bf m}_i)^T{\bf\Sigma}_i^{-1}({\bf x}-{... ...\frac{1}{2}\;log \vert{\bf\Sigma}_j \vert + log P(\omega_j)$$

is a quadric (multi-variable quadratic) equation:

$${\bf x}^T{\bf W}{\bf x}+{\bf w}^T{\bf x}+w=0$$

where ${\bf W}$ is an n by n matrix:

$${\bf W}=-\frac{1}{2}({\bf\Sigma}_i^{-1}-{\bf\Sigma}_j^{-1})$$

${\bf w}$ is an n by 1 vector:

$${\bf w}={\bf\Sigma}_i^{-1}{\bf m}_i-{\bf\Sigma}_j^{-1}{\bf m}_j$$

and $w$ is a scalar:

$$w=-\frac{1}{2}({\bf m}_i^T{\bf\Sigma}^{-1}{\bf m}_i-{\bf m}_j... ...vert{\bf\Sigma}_j\vert} +log \frac{P(\omega_i)}{P(\omega_j)}$$

These boundaries in the N-D feature space are in general quadric hyper-surfaces, such as hyper-sphere, hyper-ellipsoid, hyper-parabola, hyper-hyperbola, etc.

Now consider several special cases:

• All classes have equal a priori probability:
  
  $$P(\omega_i)=P(\omega_j)\;\;\;\;\mbox{for all i,j}$$
  
  
  then the last term of $D_i({\bf x})$ can be dropped:
  
  $$D_i({\bf x})=-\frac{1}{2}({\bf x}-{\bf m}_i)^T{\bf\Sigma}_i... ...bf x}-{\bf m}_i) -\frac{1}{2}\;log \vert{\bf\Sigma}_i \vert$$
  
  
  In particular, if ${\bf x}={\bf m}_i={\bf m}_j$, ${\bf x}$ will be classified to class $i$ if $log\;\vert{\bf\Sigma}_i\vert < \log\;\vert{\bf\Sigma}_j\vert$.
• All classes have the same covariance matrix ${\bf\Sigma}_i={\bf\Sigma}$, the discriminant function becomes:
  
  $$D_i({\bf x})=-\frac{1}{2}({\bf x}-{\bf m}_i)^T{\bf\Sigma}^{-1}({\bf x}-{\bf m}_i) +log P(\omega_i)$$
  
  
  In particular, if all $P(\omega_i)$ are the same, $D_i({\bf x})$ becomes the negative of Mahalanobis distance.

  The boundary equation $D_i({\bf x})=D_j({\bf x})$ between $\omega_i$ and $\omega_j$ becomes
  

  $$-\frac{1}{2}({\bf x}-{\bf m}_i)^T\Sigma^{-1}({\bf x}-{\bf m... ...-{\bf m}_j)^T\Sigma^{-1}({\bf x}-{\bf m}_j)+log P(\omega_j)$$
  
  
  which can be simplified to a linear equation:
  
  $${\bf w}^T{\bf x}-w=0$$
  
  
  where
  
  $${\bf w}={\bf\Sigma}^{-1}({\bf m}_i-{\bf m}_j)$$
  
  
  and
  
  $$w=-\frac{1}{2}({\bf m}_i^T{\bf\Sigma}^{-1}{\bf m}_i -{\bf ... ...f\Sigma}^{-1}{\bf m}_j)+log \frac{P(\omega_i)}{P(\omega_j)}$$
  
  
  This linear equation represents a hyper-plane between the two points ${\bf m}_i$ and ${\bf m}_j$ and perpendicular to the straight line ${\bf\Sigma}^{-1}({\bf m}_i-{\bf m}_j)$ (the straight line ${\bf m}_i-{\bf m}_j$ rotated by matrix ${\bf\Sigma}^{-1}$).

• All classes have the same isotropic distribution:
  
  $${\bf\Sigma}_i={\bf\sigma}^2 {\bf I}=diag[\sigma^2,\cdots,\sigma^2]$$
  
  
  then
  
  $$\vert{\bf\Sigma}_i\vert=\sigma^{2N}$$
  
  
  and $D_i({\bf x})$ becomes
  
  $$D_i({\bf x})=-\frac{\left\vert {\bf x}-{\bf m}_i\right\vert^2}{2\sigma^2}+log P(\omega_i)$$
  
  
  Note that the term $log \vert{\bf\Sigma}_i\vert$ has been dropped from the original expression of $D_i({\bf x})$ as it is now the same for all classes.

  The boundary equation in the feature space between $\omega_i$ and $\omega_j$ is:
  

  $$D_i({\bf x})=D_j({\bf x}),\;\;\;\;\; \frac{\left\vert {\bf... ... {\bf x}-{\bf m}_j\right\vert^2}{2\sigma^2}-log P(\omega_j)$$
  
  
  which can be simplified to a linear equation:
  
  $${\bf w}^T{\bf x}-w=0,\;\;\;\;\;\;\mbox{i.e.,}\;\;\;\;\;\;\; {\bf w}^T{\bf x}=w$$
  
  
  where ${\bf w}$ is a vector
  
  $${\bf w}={\bf m}_i-{\bf m}_j$$
  
  
  and $w$ is a scalar
  
  $$w=-({\bf m}_i^T{\bf m}_i-{\bf m}_j^T{\bf m}_j)+2\sigma^2\;log \frac{P(\omega_i)}{P(\omega_j)}$$
  
  
  This linear equation represents a hyper-plane between the two points ${\bf m}_i$ and ${\bf m}_j$ and perpendicular to the straight line passing through these points.

• Further, if all classes have the same $P(\omega_i)$, $D_i({\bf x})$ becomes
  
  $$D_i({\bf x})=-\left\vert {\bf x}-{\bf m}_i\right\vert^2 =-... ...x}-{\bf m}_i)^T({\bf x}-{\bf m}_i)=-d_2^2({\bf x},{\bf m}_i)$$
  
  
  and the Bayes classifier becomes minimum distance classifier using Euclidean distance (maximizing $D_i({\bf m})$ is equivalent to minimizing $d_2({\bf x},{\bf m}_i)$).