Vectors and their norms

The norm of vector ${\bf x} \in V$ in a vector space $V$ is a non-negative real value representing intuitively the length, size, or magnitude of the vector. Specifically, the norm of vector ${\bf x} \in V$ satisfies:

• Positivity:
  
  $$\vert\vert{\bf x}\vert\vert\ge 0,\;\;\;\;\;\;\vert\vert{\bf x}\vert\vert=0\;\;\;\mbox{iff}\;\;\;{\bf x}={\bf0}$$
  
  

• Homogeneity:
  
  $$\vert\vert a{\bf x}\vert\vert=\vert a\vert\;\vert\vert{\bf x}\vert\vert$$
  
  

• Triangle inequality:
  
  $$\vert\vert{\bf x}+{\bf y}\vert\vert\le \vert\vert{\bf x}\vert\vert+\vert\vert{\bf y}\vert\vert$$
  
  

where $a$ is a real or complex scalar. Subtracting $\vert\vert{\bf y}\vert\vert$ from both sides, and redefine ${\bf x}+{\bf y}$ as ${\bf x}$ (the original ${\bf x}$ now becomes ${\bf x}-{\bf y}$), we get

$$\vert\vert{\bf x}\vert\vert-\vert\vert{\bf y}\vert\vert\le \vert\vert{\bf x}-{\bf y}\vert\vert$$

Consider the norms in the following common vector fields. If the vector ${\bf x}=x$ is a real number in the real space $V=\mathbb{R}$, then its norm is simply its absolute value $\vert\vert{\bf x}\vert\vert=\vert x\vert$.

If the vector ${\bf z}=z=x+j\,y$ is a complex number in the complex space $V=\mathbb{C}$, then its norm is simply its modulus $\vert\vert{\bf z}\vert\vert=\vert z\vert=\sqrt{x^2+y^2}$.

If ${\bf x}=[x_1,\cdots,x_n]^T$ is a vector in an n-D vector space $V=\mathbb{R}^n$ or $V=\mathbb{C}^n$, then we can use the p-norms defined as

$$\vert\vert{\bf x}\vert\vert _p=\left(\sum_{i=1}^N \vert x_i\vert^p\right)^{1/p},\;\;\;\;\;\; 1\le p \le \infty$$

The p-norm so defined satisfies the three requirements in the definition of the vector norm. The first two are trivial to show, while the third one happens to be Minkowski's inquality:

$$\vert\vert{\bf x}+{\bf y}\vert\vert _p\le \vert\vert{\bf x}\vert\vert _p+\vert\vert{\bf y}\vert\vert _p$$

The commonly used p-norms are for $p=1$, $p=2$, and $p=\infty$:

• The absolute sum of all $N$ elements:
  
  $$\vert\vert{\bf x}\vert\vert _1=\sum_{i=1}^N \vert x_i\vert$$
  
  

• Euclidean norm:
  
  $$\vert\vert{\bf x}\vert\vert _2=\sqrt{{\bf x}^T{\bf x}}=\sqrt{\sum_{i=1}^N x_i^2}$$
  
  

• The maximum absolute value of all $N$ elements:
  
  $$\vert\vert{\bf x}\vert\vert _\infty=\max\{ x_1,\cdots, x_N \}$$
  
  

Out of the three vector norms, the Euclidean 2-norm represents the geometric length of a vector in 2 or 3-D space, which is conserved, or invariant, under vector rotation, i.e., ${\bf y}={\bf R}{\bf x}$ with an orthogonal (unitary if in complex field) matrix ${\bf R}^T={\bf R}^{-1}$ satisfying ${\bf R}^T{\bf R}={\bf R}^{-1}{\bf R}={\bf I}$.

$$\vert\vert{\bf y}\vert\vert _2^2={\bf y}^T{\bf y}=({\bf R}{\... ...bf R}{\bf x}={\bf x}^T{\bf x}=\vert\vert{\bf x}\vert\vert _2^2$$

i.e., $\vert\vert{\bf y}\vert\vert=\vert\vert{\bf x}\vert\vert$, rotation does not change the length of a vector.

The distance between two points ${\bf x}, {\bf y}\in V$ in a vector space is defined as the norm of the difference ${\bf x}-{\bf y}$.

$$d_p({\bf x},{\bf y})=\vert\vert{\bf x}-{\bf y}\vert\vert _p ... ...N \vert x_i-y_i\vert^p\right)^{1/p},\;\;\;\;(1\le p\le \infty)$$

• $p=1$ (city block or Mahatan distance):
  
  $$d_1({\bf x},{\bf y})=\vert\vert{\bf x}-{\bf y}\vert\vert _1=\sum_{i=1}^N \vert x_i-y_i\vert$$
  
  

• $p=2$ (Euclidean distance):
  
  $$d_2({\bf x},{\bf y})=\vert\vert{\bf x}-{\bf y}\vert\vert _2=\sqrt{\sum_{i=1}^N \vert x_i-y_i\vert^2}$$
  
  

• $p=\infty$ (Chebyshev distance):
  
  $$d_\infty({\bf x},{\bf y})=\vert\vert{\bf x}-{\bf y}\vert\vert _\infty=\max_i \vert x_i-y_i\vert$$
  
  

The three curves or surfaces correspond to the iso-distances to the center point, formed by all points with equal distances to the point at the center, blue for $d_\infty$, black for $d_2$, and red for $d_1$.

Two different vectors ${\bf x}$ and ${\bf y}$ are orthogonal to each other if their inner product is zero $<{\bf x},{\bf y}>=0$. If they are also normalized with $\vert\vert{\bf x}\vert\vert=\vert\vert{\bf y}\vert\vert=1$, then they are orthonormal,

$$<{\bf x},{\bf y}>=\delta_{xy} \stackrel{\triangle}{=}\left\{... ...bf x}={\bf y} \\ 0 & {\bf x} \neq {\bf y} \end{array} \right.$$