Pseudo-inverse

Let ${\bf A}$ be an $m\times n$ matrix. If $m\ne n$, then ${\bf A}$ does not have an inverse. However, we can find its pseudo-inverse ${\bf A}^-$, an $n\times m$ matrix, as shown below.

• If ${\bf A}$ has more rows than columns; i.e., $m>n$, then
  
  $${\bf A}^-=({\bf A}^T{\bf A})^{-1}{\bf A}^T$$
  
  
  We can verify that ${\bf A}^-{\bf A}={\bf I}$:
  
  $${\bf A}^-{\bf A}=({\bf A}^T{\bf A})^{-1}{\bf A}^T{\bf A}={\bf I}_{n\times n}$$
  
  
  But ${\bf A}{\bf A}^-\ne{\bf I}$.
• If ${\bf A}$ has more columns than rows; i.e., $m<n$, then
  
  $${\bf A}^-={\bf A}^T({\bf A}{\bf A}^T)^{-1}$$
  
  
  We can verify that ${\bf A}{\bf A}^-={\bf I}$:
  
  $${\bf A}{\bf A}^-={\bf A}{\bf A}^T({\bf A}{\bf A}^T)^{-1}={\bf I}_{m\times m}$$
  
  
  But ${\bf A}^-{\bf A}\ne{\bf I}$.

Note that the two pseudo-inverses defined above for $m>n$ and $m<n$ are essentially the same. To see this, we first assume ${\bf A}$ has more rows than columns ($m>n$), then another matrix defined as ${\bf B}={\bf A}^T$ has more columns than rows. Taking the transpose on both sides of the first pseudo inverse above, we get

$$({\bf A}^-)^T=[({\bf A}^T{\bf A})^{-1}{\bf A}^T]^T ={\bf A}({\bf A}^T{\bf A})^{-1}=({\bf A}^T)^{-};$$

i.e.,

$${\bf B}^-={\bf B}^T({\bf B}{\bf B}^T)^{-1}$$

which is the same as the second definition of the pseudo inverse.

We can also show that $({\bf A}^-)^-={\bf A}$. If $m>n$, then we have

$$({\bf A}^-)^- = [({\bf A}^T{\bf A})^{-1}{\bf A}^T]^- =[({\bf A}^T{\bf A})^{-1}{\... ...{\bf A}^T{\bf A})^{-1}{\bf A}^T[({\bf A}^T{\bf A})^{-1}{\bf A}^T]^T\right]^{-1}$$

$$= {\bf A}({\bf A}^T{\bf A})^{-1}\left[ ({\bf A}^T{\bf A})^{-1}{\bf A}^T{\bf A}({\bf A}^T{\bf A})^{-1}\right]^{-1}$$

$$= {\bf A}({\bf A}^T{\bf A})^{-1}({\bf A}^T{\bf A})={\bf A}.$$

Similarly. we can show the same is true if $m<n$. In particular, when $m=n$, ${\bf A}$ is invertible and both versions of the pseudo-inverse defined above become the regular inverse ${\bf A}^-={\bf A}^{-1}$.