Inner Product Space

• Vector space

  A vector space is a set $V$ with two operations of addition and scalar multiplication defined for its members, referred to as vectors.

  1. Vector addition maps any two vectors ${\bf x}, {\bf y}\in V$ to another vector ${\bf x}+{\bf y} \in V$ satisfying the following properties:
     • Commutativity: ${\bf x}+{\bf y}={\bf y}+{\bf x}$.
     • Associativity: ${\bf x}+({\bf y}+{\bf z})=({\bf x}+{\bf y})+{\bf z}$.
     • Existence of zero: there is a vector ${\bf0} \in V$ such that: ${\bf0}+{\bf x}={\bf x}+{\bf0}={\bf x}$.
     • Existence of inverse: for any vector ${\bf x} \in V$, there is another vector $-{\bf x}\in V$ such that ${\bf x}+(-{\bf x})={\bf0}$.
  2. Scalar multiplication maps a vector ${\bf x} \in V$ and a real or complex scalar $a\in C$ to another vector $a{\bf x} \in V$ with the following properties:
     • $a({\bf x}+{\bf y})=a{\bf x}+a{\bf y}$.
     • $(a+b){\bf x}=a{\bf x}+b{\bf x}$.
     • $ab{\bf x}=a(b{\bf x})$.
     • $1{\bf x}={\bf x}$.

  Listed below is a set of typical vector spaces for various types of signal of interest.

  • $N$-D vector space $\mathbb{R}^N$ or $\mathbb{C}^N$

    This space contains all $N$-D vectors expressed as an $N$-tuple, an ordered list of $N$ elements (or components):
    

    $${\bf x}=\left[ \begin{array}{c}x_1 x_2 \vdots x_N \end{array} \right] =[x_1,x_2,\ldots,x_N]^T,$$
    
    
    which can be used to represent a discrete signal containing $N$ samples. We will always represent a vector as a column vector, or the transpose of a row vector. The space is denoted by either $\mathbb{C}^N$ if the elements are complex $x_n\in \mathbb{C}$, or $\mathbb{R}^N$ if they are all real $x_n\in \mathbb{R}$ ($n=1,\ldots,N$).

  • A vector space can be defined to contain all $M\times N$ matrices composed of $N$ $M$-D column vectors:
    
    $${\bf A}=[{\bf a}_1,\ldots,{\bf a}_N] =\left[ \begin{array}... ...\ x_{M,1} & x_{M,2} & \cdots & x_{M,N} \end{array} \right],$$
    
    
    where the $n$th column is an $M$-D vector ${\bf a}_n=[x_{1,n},\ldots,x_{M,n}]^T$. Such a matrix can be converted to an $MN$-D vector by cascading all of the column (or row) vectors.

  • $l^2$ space:

    The dimension $N$ of $\mathbb{R}^N$ or $\mathbb{C}^N$ can be extended to infinity so that a vector in the space becomes a sequence ${\bf x}=[\ldots,x_n,\ldots]^T$ for $0\le n< \infty$ or $-\infty<n<\infty$. If all vectors are square-summable, the space is denoted by $l^2$. All discrete energy signals are vectors in $l^2$.

  • ${\cal L}^2$ space:

    A vector space can also be a set of real or complex valued continuous functions $x(t)$ defined over either a finite range such as $0 \le t < T$, or an infinite range $-\infty<t<\infty$. If all functions are square-integrable, the space is denoted by ${\cal L}^2$. All continuous energy signals are vectors in ${\cal L}^2$.

  Note that the term ``vector'', generally denoted by ${\bf x}$, may be interpreted in two different ways. First, in the most general sense, it represents a member of a vector space, such as any of the vector spaces considered above; e.g., a function ${\bf x}=x(t) \in {\cal L}^2$. Second, in a more narrow sense, it can also represent a tuple of $N$ elements, an $N$-D vector ${\bf x}=[x_1,\ldots,x_N]^T\in \mathbb{C}^N$, where $N$ may be infinity. It should be clear what a vector ${\bf x}$ represents from the context.

• Inner product

  An inner product in a vector space $V$ is a function that maps two vectors ${\bf x}, {\bf y}\in V$ to a scalar $\langle {\bf x},{\bf y}\rangle \in \mathbb{C}$ and satisfies the following conditions:

  • Positive definiteness:
    
    $$\langle {\bf x},{\bf x}\rangle \;\ge\; 0,\;\;\;\;\;\;\;\; ... ...{\bf x},{\bf x}\rangle =0\;\;\; \mbox{iff}\;\;{\bf x}={\bf0}.$$
    
    

  • Conjugate symmetry:
    
    $$\langle {\bf x},{\bf y}\rangle=\overline{\langle {\bf y},{\bf x}\rangle}.$$
    
    
    If the vector space is real, the inner product becomes symmetric:
    
    $$\langle {\bf x},{\bf y}\rangle=\langle {\bf y},{\bf x}\rangle.$$
    
    

  • Linearity in the first variable:
    
    $$\langle a{\bf x}+b{\bf y},{\bf z}\rangle=a\langle {\bf x},{\bf z}\rangle+b\langle {\bf y},{\bf z}\rangle,$$
    
    
    where $a,b\in \mathbb{C}$. The linearity does not apply to the second variable:
    

    $$\langle {\bf x}, a{\bf y}+b{\bf z}\rangle=\overline{\langle a{\bf... ...e} =\overline{a\langle {\bf y},{\bf x}\rangle+b\langle {\bf z},{\bf x}\rangle}$$

    $$= \overline{a}\langle {\bf x},{\bf y}\rangle+\overline{b}\langle {\... ... z}\rangle \ne a\langle {\bf x},{\bf y}\rangle+b\langle {\bf x},{\bf z}\rangle$$

    
    
    unless the coefficients are real $a,b \in \mathbb{R}$. As a special case, when $b=0$, we have
    
    $$\langle a{\bf x},{\bf y}\rangle=a\langle {\bf x},{\bf y}\ra... ...},a{\bf y}\rangle=\overline{a}\langle {\bf x},{\bf y}\rangle.$$
    
    
    More generally we have
    
    $$\left\langle \sum_n c_n {\bf x}_n,{\bf y}\right\rangle=\sum... ...angle=\sum_n \overline{c}_n \langle {\bf x},{\bf y}_n\rangle.$$
    
    

  Some examples of the inner product are listed below:

  • In an N-D vector space, the inner product, also called the dot product, of two vectors ${\bf x}=[x_1,\ldots,x_N]^T$ and ${\bf y}=[y_1,\ldots,y_N]^T$ is defined as
    
    $$\langle {\bf x}, {\bf y}\rangle={\bf x}^T\overline{{\bf y}}... ...ne{y}_N \end{array} \right]=\sum_{n=1}^N x_n \overline{y}_n,$$
    
    
    where ${\bf y}^*=\overline{{\bf y}}^T$ is the conjugate transpose of ${\bf y}$.

  • In a space of 2-D matrices containing $M\times N$ elements, the inner product of two matrices ${\bf A}$ and ${\bf B}$ is defined as
    
    $$\langle {\bf A}, {\bf B}\rangle=\sum_{m=1}^M\sum_{n=1}^N a_{m,n}\overline{b}_{m,n}.$$
    
    
    When the column (or row) vectors of ${\bf A}$ and ${\bf B}$ are concatenated to form two $MN$-D vectors, their inner product takes the same form as that of two N-D vectors.

  • In a function space, the inner product of two function vectors ${\bf x}=x(t)$ and ${\bf y}=y(t)$ is defined as
    
    $$\langle x(t),y(t)\rangle=\int_a^b x(t)\overline{y(t)} \;dt ... ...verline{x(t)}y(t)} \;dt =\overline{\langle y(t),x(t)\rangle}.$$
    
    

  • The inner product of two random variables $x$ and $y$ can be defined as
    
    $$\langle x,y\rangle=E[x\overline{y}].$$
    
    
    If the two random variables have zero means; i.e., $\mu_x=E(x)=0$ and $\mu_x=E(x)=0$, the inner product above is also their covariance:
    
    $$\sigma^2_{xy}=E[(x-\mu_x)\overline{(y-\mu_y)}]= E(x\overli... ...})-\mu_x\overline{\mu}_y=E(x\overline{y})=\langle x,y\rangle.$$
    
    

• Vector norm

  The norm of a vector ${\bf x} \in V$ is defined below as a certain measurement of its size or length:
  

  $$\vert\vert{\bf x}\vert\vert=\sqrt{\langle {\bf x},{\bf x}\r... ...\vert\vert{\bf x}\vert\vert^2=\langle {\bf x},{\bf x}\rangle.$$
  
  
  The norm $\vert\vert{\bf x}\vert\vert$ is non-negative and it is zero if and only if ${\bf x}={\bf0}$. In particular, if $\vert\vert{\bf x}\vert\vert=1$, then it is said to be normalized and becomes a unit vector. Any vector can be normalized when divided by its own norm: ${\bf x}/\vert\vert{\bf x}\vert\vert$. The vector norm squared $\vert\vert{\bf x}\vert\vert^2=\langle {\bf x},{\bf x}\rangle$ can be considered as the energy of the vector.

  In particular, in an N-D unitary space, the norm of a vector ${\bf x}=[x_1,\ldots,x_N]^T\in \mathbb{C}^N$ is
  

  $$\vert\vert{\bf x}\vert\vert=\sqrt{\langle {\bf x},{\bf x}\r... ...t)^{1/2} =\left(\sum_{n=1}^N \vert x_n\vert^2\right)^{1/2}.$$
  
  
  The total energy contained in this vector is its norm squared:
  
  $${\cal E}=\vert\vert{\bf x}\vert\vert^2=\langle {\bf x},{\bf x}\rangle=\sum_{n=1}^N \vert x_n\vert^2.$$
  
  

  Similarly, in a function space, the norm of a function vector ${\bf x}=x(t)$ is defined as
  

  $$\vert\vert{\bf x}\vert\vert=\left( \int_a^b x(t)\overline{x... .../2} =\left( \int_a^b \vert x(t)\vert^2\; \;dt \right)^{1/2},$$
  
  
  where the lower and upper integral limits $a<b$ are two real numbers, which may be extended to all real values $\mathbb{R}$ in the entire real axis $-\infty<t<\infty$. This norm exists only if the integral converges to a finite value; i.e., $x(t)$ is an energy signal containing finite energy:
  
  $$\int_{-\infty}^\infty \big\vert x(t)\big\vert^2 \;dt < \infty.$$
  
  
  All such functions $x(t)$ satisfying the above are square-integrable, and they form a function space denoted by ${\cal L}^2(\mathbb{R})$.

• Cauchy-Schwarz inequality

  The Cauchy-Schwarz inequality holds for any two vectors ${\bf x}, {\bf y}\in V$ in an inner product space $V$:
  

  $$\vert\langle {\bf x},{\bf y}\rangle\vert^2 \; \le\; \langle... ...e \vert\vert{\bf x}\vert\vert \; \vert\vert{\bf y}\vert\vert.$$
  
  

  Proof: If either ${\bf x}$ or ${\bf y}$ is zero, $\langle{\bf x},{\bf y}\rangle=0$, the theorem holds (an equality). Otherwise, we consider the following inner product:
  

  $$\langle {\bf x}-\lambda{\bf y}, {\bf x}-\lambda{\bf y}\rang... ...angle+\vert\lambda\vert^2\vert\vert{\bf y}\vert\vert^2 \ge 0,$$
  
  
  where $\lambda\in \mathbb{C}$ is an arbitrary complex number, which can be assumed to be:
  
  $$\lambda=\frac{\langle {\bf x},{\bf y}\rangle}{\vert\vert{\b... ...\bf x},{\bf y}\rangle\vert^2}{\vert\vert{\bf y}\vert\vert^4}.$$
  
  
  Substituting these into the previous equation we get
  
  $$\vert\vert{\bf x}\vert\vert^2-\frac{\vert\langle {\bf x},{\... ...\le \vert\vert{\bf x}\vert\vert\;\vert\vert{\bf y}\vert\vert.$$
  
  
  The equation holds only if ${\bf x}-\lambda{\bf y}=0$ or ${\bf x}=\lambda{\bf y}$, i.e., the two vectors are linearly dependent.

• Distance between two vectors

  The distance $d({\bf x},{\bf y})$ between two vectors ${\bf x}$ and ${\bf y}$ is a real constant that satisfies:

  • $d({\bf x},{\bf y})$ iff ${\bf x}={\bf y}$
  • $d({\bf x},{\bf y})=d({\bf y},{\bf x})$
  • $d({\bf x},{\bf y})\le d({\bf x},{\bf z})+d({\bf z},{\bf y})$
  In an inner-product space in which the inner product $\langle{\bf x},{\bf y}\rangle$ between any two vectors ${\bf x}$ and ${\bf y}$ is defined, the distance between the two points is $\langle({\bf x}-{\bf y}),({\bf x}-{\bf y})\rangle$.

• Angle between two vectors

  The angle between two vectors ${\bf x}$ and ${\bf y}$ is defined as
  

  $$\theta=\cos^{-1} \left( \frac{\langle {\bf x},{\bf y}\rangl... ...vert{\bf x}\vert\vert\;\vert\vert{\bf y}\vert\vert} \right).$$
  
  
  Now the inner product of ${\bf x}$ and ${\bf y}$ can also be written as
  
  $$\langle {\bf x},{\bf y}\rangle=\vert\vert{\bf x}\vert\vert\... ... \le\vert\vert{\bf x}\vert\vert\;\vert\vert{\bf y}\vert\vert$$
  
  
  In particular,
  • if $\theta=0$, $\cos\theta=1$, then ${\bf x}$ and ${\bf y}$ are collinear or linearly dependent, and the inner product is maximized:
    
    $$\langle {\bf x},{\bf y}\rangle=\vert\vert{\bf x}\vert\vert\;\vert\vert{\bf y}\vert\vert$$
    
    
    i.e., the Cauchy-Schwarz inequality becomes an equality.

  • if $0<\theta<\pi/2$, $0<\cos\theta<1$, we get the Cauchy-Schwarz inequality:
    
    $$\langle {\bf x},{\bf y}\rangle <\vert\vert{\bf x}\vert\vert\;\vert\vert{\bf y}\vert\vert$$
    
    

  • if $\theta=\pi/2$, $\cos\theta=0$, then ${\bf x}$ and ${\bf y}$ are orthogonal to each other, and the inner product is minimized:
    
    $$\langle {\bf x},{\bf y}\rangle=0$$
    
    

• Orthogonality

  Two vectors ${\bf x}$ and ${\bf y}$ are orthogonal or perpendicular to each other, denoted by ${\bf x}\perp{\bf y}$, if their inner product is zero $\langle{\bf x},{\bf y}\rangle=0$, i.e., the angle between them is
  

  $$\theta=\cos^{-1} 0=\frac{\pi}{2}$$
  
  

• Projection

  The orthogonal projection of a vector ${\bf x} \in V$ onto another vector ${\bf y}\in V$ is defined as a vector
  

  $${\bf p}_{{\bf y}}({\bf x}) =\frac{\langle{\bf x},{\bf y}\r... ...\vert \cos\theta \frac{{\bf y}}{\vert\vert{\bf y}\vert\vert},$$
  
  
  Note that
  
  $$\vert\vert{\bf p}_{{\bf y}}({\bf x})\vert\vert=\vert\vert{\bf x}\vert\vert \cos\theta$$
  
  

• Inner product space

  An inner product space is a vector space with inner product defined. In particular, when the inner product is defined, $\mathbb{C}^N$ is called a unitary space and $\mathbb{R}^N$ is called a Euclidean space.

• Metric space

  A metric space is a vector space $V$ in which the metric or distance $d({\bf x},{\bf y})$ between any two vector (two points) ${\bf x}$ and ${\bf y}$ is defined.

• Cauchy space

  A sequence of points in a metric space $x_0, x_1, x_2,\cdots$ is a Cauchy sequence if it converges, i.e., for any $\epsilon>0$, there exists an integer $N>0$ so that the following is true for any $m,n>N$:
  

  $$d(x_m,x_n)<\epsilon$$
  
  
  If the limit of any Cauchy sequence of points in the space is also in the space, the space is complete, referred to as a Cauchy space.