Matrix norms

An $M\times N$ matrix ${\bf A}$ can be considered as a particular kind of vector ${\bf x}={\bf A}\in R^{m,n}$, and its norm is any function that maps ${\bf A}$ to a real number $\vert\vert{\bf A}\vert\vert$ that satisfies the following required properties:

• Positivity:
  
  $$\vert\vert{\bf A}\vert\vert\ge 0,\;\;\;\;\;\;\vert\vert{\bf A}\vert\vert=0\;\;\;\mbox{iff}\;\;\;{\bf A}={\bf0}$$
  
  

• Homogeneity:
  
  $$\vert\vert a{\bf A}\vert\vert=\vert a\vert\;\vert\vert{\bf A}\vert\vert$$
  
  

• Triangle inequality:
  
  $$\vert\vert{\bf A}+{\bf B}\vert\vert\le \vert\vert{\bf A}\ve... ...\vert{\bf B}\vert\vert\le \vert\vert{\bf A}-{\bf B}\vert\vert$$
  
  

In addition to the three required properties for matrix norm, some of them also satisfy these additional properties not required of all matrix norms:

• Subordinance:
  
  $$\vert\vert{\bf A}{\bf x}\vert\vert\le \vert\vert{\bf A}\vert\vert \cdot \vert\vert{\bf x}\vert\vert$$
  
  

• Submultiplicativity:
  
  $$\vert\vert{\bf A}{\bf B}\vert\vert\le \vert\vert{\bf A}\vert\vert \cdot \vert\vert{\bf B}\vert\vert$$
  
  

We now consider some commonly used matrix norms.

• Element-wise norms

  If we treat the $M\times N$ elements of ${\bf A}$ are the elements of an $MN$-dimensional vector, then the p-norm of this vector can be used as the p-norm of ${\bf A}$:
  

  $$\vert\vert{\bf A}\vert\vert _p=\left\{ \sum_{i=1}^M\sum_{j=1}^N \vert a_{ij}\vert^p\right\}^{1/p}$$
  
  
  Specially, we consider the following three cases for $p=1,\;2,\;\infty$.
  • $\vert\vert{\bf A}\vert\vert _1$ is the absolute sum of all elements of ${\bf A}$:
    
    $$\vert\vert{\bf A}\vert\vert _1=\sum_{i=1}^M\sum_{j=1}^N \vert a_{ij}\vert$$
    
    

  • $\vert\vert{\bf A}\vert\vert _\infty$ is the maximum norm, the maximum absolute value among all $MN$ elements of ${\bf A}$:
    
    $$\vert\vert{\bf A}\vert\vert _\infty=\max\{ \vert a_{ij}\vert,\;\; 1\le i\le M,\;1\le j\le N \}$$
    
    

  • $\vert\vert{\bf A}\vert\vert _2$ is the Frobenius norm
    
    $$\vert\vert{\bf A}\vert\vert _2=\vert\vert{\bf A}\vert\vert ... ...sqrt{\sum_{i=1}^R \lambda_i} =\sqrt{\sum_{i=1}^R \sigma_i^2}$$
    
    
    where $R\le \min(M,N)$ is the rank of ${\bf A}$, $\lambda_i=\sigma_2^2$ is the ith non-zero eigenvalues of ${\bf A}^*{\bf A}$, and $\sigma_i=\sqrt{\lambda_i}$ the ith singular value of ${\bf A}$ ($i=1,\cdots,R$). (This Frobenius norm $\vert\vert{\bf A}\vert\vert _2$ is implemented in Matlab by the function norm(A,'fro').)

  The Frobenius norm of a unitary (orthogonal if real) matrix ${\bf R}$ satisfying ${\bf R}^*={\bf R}^{-1}$ or ${\bf R}^*{\bf R}={\bf R}^{-1}{\bf R}={\bf I}$ is:
  
  $$\vert\vert{\bf R}\vert\vert _F^2= tr({\bf R}^*{\bf R})=tr {\bf I}=N$$
  
  
  The Frobenius norm is the only one out of the above three matrix norms that is unitary invariant, i.e., it is conserved or invariant under a unitary transformation (such as a rotation) ${\bf B}={\bf R}^*{\bf A}{\bf R}$:
  

  $$\vert\vert{\bf B}\vert\vert _F^2 = tr\left({\bf B}^*{\bf B}\right) =tr\left[\left({\bf R}^*{\bf A}{... ...ght)\right] =tr \left( {\bf R}^*{\bf A}^*{\bf R}{\bf R}^*{\bf A}{\bf R}\right)$$

  $$= tr \left( {\bf R}^*{\bf A}^*{\bf A}{\bf R}\right) =tr \left( {\b... ...f A}\right)=\vert\vert{\bf A}\vert\vert _F^2 =\vert\vert{\bf A}\vert\vert _F^2$$

  
  
  where we have used the property of the trace $tr\left({\bf A}{\bf B}\right)=tr\left({\bf B}{\bf A}\right)$.

• Induced or operator norms

  $\vert\vert{\bf A}\vert\vert$ of a matrix ${\bf A}$ is based on any vector norm $\vert\vert{\bf x}\vert\vert$
  

  $$\vert\vert{\bf A}\vert\vert=\sup_{\vert\vert{\bf x}\vert\ve... ...vert{\bf A}{\bf x}\vert\vert / \vert\vert{\bf x}\vert\vert \}$$
  
  
  ($\vert\vert{\bf A}\vert\vert$ is sub-ordinate to the vector norm $\vert\vert{\bf x}\vert\vert$.) Here $\sup_x f(x)$ is supremum of $f(x)$, which is the same as the maximum $\max_x f(x)$ if the function is closed and bounded. Otherwise, the maximum does not exist and the supremum is the least upper bound of the function.

  Note that the norm of the identity matrix ${\bf A}={\bf I}$ is
  

  $$\vert\vert{\bf I}\vert\vert=\sup_{\vert\vert{\bf x}\vert\ve... ...\vert{\bf x}\vert\vert=1} \{ \vert\vert{\bf x}\vert\vert \}=1$$
  
  

  We now prove the matrix norm defined above satisfy all properties given above. (Recall $\vert a+b\vert\le \vert a\vert+\vert b\vert$, $\vert ab\vert\le \vert a\vert\;\vert b\vert$.)

  1. $\vert\vert{\bf A}\vert\vert>0$ if ${\bf A}\ne {\bf0}$, this is trivially obvious.
  2. $\vert\vert a{\bf A}\vert\vert = \vert a\vert \vert\vert{\bf A}\vert\vert$
     
     $$\vert\vert a{\bf A}\vert\vert=\sup_{\vert\vert{\bf x}\vert\... ...f x}\vert\vert \} =\vert a\vert\;\vert\vert{\bf A}\vert\vert$$
     
     

  3. $\vert\vert{\bf A}+{\bf B}\vert\vert\le \vert\vert{\bf A}\vert\vert+\vert\vert{\bf B}\vert\vert$
     

     $$\vert\vert{\bf A}+{\bf B}\vert\vert = \sup_{\vert\vert{\bf x}\vert\vert=1} \{ \vert\vert({\bf A}+{\bf B... ...=1} \{ \vert\vert{\bf A}{\bf x}\vert\vert+\vert\vert{\bf B}{\bf x}\vert\vert \}$$

     $$\le \sup_{\vert\vert{\bf x}\vert\vert=1} \{ \vert\vert{\bf A}{\bf x}\... ...}{\bf x}\vert\vert \} =\vert\vert{\bf A}\vert\vert+\vert\vert{\bf B}\vert\vert$$

     
     

  4. $\vert\vert{\bf A}{\bf x}\vert\vert\le \vert\vert{\bf A}\vert\vert\;\vert\vert{\bf x}\vert\vert$
     
     $$\vert\vert{\bf A}\vert\vert\;\vert\vert{\bf x}\vert\vert=\v... ...t\;\vert\vert{\bf Ay}\vert\vert / \vert\vert{\bf y}\vert\vert$$
     
     
     As ${\bf y}$ is arbitrary, we let ${\bf y}={\bf x}$ and get
     
     $$\vert\vert{\bf A}\vert\vert\;\vert\vert{\bf x}\vert\vert\ge \vert\vert{\bf Ax}\vert\vert$$
     
     

  5. $\vert\vert{\bf A}{\bf B}\vert\vert\le \vert\vert{\bf A}\vert\vert\;\vert\vert{\bf B}\vert\vert$
     
     $$\vert\vert{\bf A}{\bf B}\vert\vert=\sup_{\vert\vert{\bf x}\... ...} = \vert\vert{\bf A}\vert\vert\;\vert\vert{\bf B}\vert\vert$$
     
     

  Specifically, the matrix p-norm $\vert\vert{\bf A}\vert\vert _p$ can be based on the vector p-norm $\vert\vert{\bf x}\vert\vert _p$, as defined in the following for $p=1, 2, \infty$.

  • When $p=1$, $\vert\vert{\bf A}\vert\vert _1$ is maximum absolute column sum:
    
    $$\vert\vert{\bf A}\vert\vert _1=\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert$$
    
    
    In Matlab this norm is implemented by the function norm(A,1).

    Proof: The 1-norm of vector ${\bf x}$ is $\vert\vert{\bf x}\vert\vert _1=\sum_{j=1}^N \vert x_j\vert$, we have
    

    $$\vert\vert{\bf A}{\bf x}\vert\vert _1 = \sum_{i=1}^M\bigg\vert\sum_{j=1}^Na_{ij}x_j\bigg\vert \le \sum_{... ...t) =\sum_{j=1}^N \left( \vert x_j\vert\;\sum_{i=1}^M \vert a_{ij}\vert \right)$$

    $$\le \left(\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert\right) ... ... x_j\vert\right)=\vert\vert{\bf A}\vert\vert _1\;\vert\vert{\bf x}\vert\vert _1$$

    
    
    Assuming the kth column of ${\bf A}$ has the maximum absolute sum and ${\bf x}$ is normalized (as required in the definition) with $\vert\vert{\bf x}\vert\vert=1$, we have
    
    $$\vert\vert{\bf A}\vert\vert _1=\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert=\sum_{i=1}^M\vert a_{ik}\vert$$
    
    
    and
    
    $$\vert\vert{\bf A}{\bf x}\vert\vert _1\le \vert\vert{\bf A}\... ...;\vert\vert{\bf x}\vert\vert _1=\sum_{i=1}^M\vert a_{ik}\vert$$
    
    
    Now we show that the equality of the above can be achieved, i.e., $\vert\vert{\bf Ax}\vert\vert _1$ is maximized, if we choose ${\bf x}={\bf e}_k=[0,\cdots,1,\cdots,0]^T$, the kth unit vector (normalized):
    
    $$\vert\vert{\bf A}{\bf x}\vert\vert _1=\sum_{i=1}^M\bigg\ver... ... a_{ik}\vert=\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert$$
    
    
    i.e., ${\bf x}={\bf e}_k$ is the vector among all other normalized vectors that maximizes $\vert\vert{\bf Ax}\vert\vert$ as required in the definition, and the resulting maximum $\vert\vert{\bf A}\vert\vert _1$ is indeed $\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert$. We therefore have
    
    $$\vert\vert{\bf A}\vert\vert _1=\sup_{\vert\vert{\bf x}\vert... ...\vert _1\} =\max_{1\le j\le N} \sum_{i=1}^M\vert a_{ij}\vert$$
    
    

  • When $p=\infty$, $\vert\vert{\bf A}\vert\vert _\infty$ is maximum absolute row sum:
    
    $$\vert\vert{\bf A}\vert\vert _\infty=\max_{1\le i\le M} \sum_{j=1}^N\vert a_{ij}\vert$$
    
    
    In Matlab this norm is implemented by the function norm(A,inf).

    Proof: When $p=\infty$, ${\bf x}$ is normalized if $\vert\vert{\bf x}\vert\vert _\infty=\max \{x_1,\cdots,x_n\}=1$. The norm of vector ${\bf Ax}$ is:
    

    $$\vert\vert{\bf Ax}\vert\vert _\infty=\max\left\{ \sum_{j=1}... ...ij}x_j\vert,\cdots, \sum_{j=1}^N\vert a_{Mj}x_j\vert\right\}$$
    
    
    which can be maximized by any normalized vector with $\vert\vert{\bf x}\vert\vert _\infty=\max \{x_1,\cdots,x_n\}=1$ to become
    
    $$\vert\vert{\bf Ax}\vert\vert _\infty=\max\left\{ \sum_{j=1}... ...rt a_{ij}\vert,\cdots, \sum_{j=1}^N\vert a_{Mj}\vert\right\}$$
    
    
    We therefore have
    
    $$\vert\vert{\bf A}\vert\vert _\infty=\sup_{\vert\vert{\bf x}... ...Ax}\vert\vert _\infty\} =\max_i\sum_{j=1}^N\vert a_{ij}\vert$$
    
    

  • When $p=2$, $\vert\vert{\bf A}\vert\vert _2$ is the spectral norm, the greatest singular value of ${\bf A}$, which is the square root of the greatest eigenvalue of ${\bf A}^*{\bf A}$, i.e., its spectral radius $\rho({\bf A}^*{\bf A})=\lambda_{max}({\bf A}^*{\bf A})$,
    
    $$\vert\vert{\bf A}\vert\vert _2=\sigma_{max}({\bf A})=\sqrt{\lambda_{max}({\bf A}^*{\bf A})} =\sqrt{\rho({\bf A}^*{\bf A})}$$
    
    
    where $\lambda_{max}({\bf A}^*{\bf A})=\sigma_{max}^2({\bf A})$ is the maximal eigenvalue of ${\bf A}^*{\bf A}$, and $\sigma_{max}({\bf A})=\sqrt{\lambda({\bf A}^*{\bf A})}$ the maximal singular value of ${\bf A}$. In Matlab this norm is implemented by the function norm(A,2) or simply norm(A).)

    Proof: When $p=2$, $\vert\vert{\bf x}\vert\vert=({\bf x}^*{\bf x})^{1/2}$, and we have
    

    $$\vert\vert{\bf Ax}\vert\vert _2 = \left[ ({\bf Ax})^*({\bf Ax})\right]^{1/2} =\left( {\bf x}^*{\bf... ...ht)^{1/2} =\left({\bf x}^*{\bf\Phi}^*{\bf\Lambda}{\bf\Phi}{\bf x}\right)^{1/2}$$

    $$= \left({\bf y}^*{\bf\Lambda}{\bf y}\right)^{1/2} =\left( \sum_{i=1}^N y_i^2\lambda_i \right)^{1/2}$$

    
    
    Here we have used the eigen-decomposition ${\bf A}^*{\bf A}={\bf\Phi}^*{\bf\Lambda}{\bf\Phi}$ of ${\bf A}^*{\bf A}$, where
    
    $${\bf\Lambda}=diag(\lambda_1,\cdots,\lambda_n), \;\;\;\;\mbox{and}\;\;\;\;\;\; {\bf\Phi}=[{\bf\phi}_1,\cdots,{\bf\phi}_n]$$
    
    
    are the diagonal eigenvalue matrix and the eigenvector matrix of ${\bf A}^*{\bf A}$, satisfying
    
    $${\bf A}^*{\bf A}{\bf\phi}=\lambda_i{\bf\phi}_i,\;\;\;(i=1,\... ...\;\;\;\;\;\; {\bf A}^*{\bf A}{\bf\Phi}={\bf\Phi}{\bf\Lambda}$$
    
    
    As ${\bf A}^*{\bf A}$ is a symmetric positive definite square matrix, all of its eigenvalues are real and positive and assumed to be sorted
    
    $$\lambda_1=\lambda_{max}\ge \lambda_2\ge \cdots \ge \lambda_n \ge 0$$
    
    
    and all corresponding eigenvectors are orthogonal and assumed to be normalized, i.e., ${\bf\phi}_i^*{\bf\phi}_j=\delta_{ij}$, or ${\bf\Phi}^*={\bf\Phi}^{-1}$ is a unitary (orthogonal if real) matrix. In the equation above, we have introduced a new vector ${\bf y}={\bf\Phi}{\bf x}$ as a unitary transform of ${\bf x}$. ${\bf y}$ can be considered as a rotated version of ${\bf x}$ with its Euclidean 2-norm conserved, $\vert\vert{\bf y}\vert\vert _2=\vert\vert{\bf x}\vert\vert _2$.

    The right-hand side of the equation above is a weighted average of the $N$ eigenvalues $\lambda_1,\cdots,\lambda_n$, which is maximized if they are weighted by a normalized vector ${\bf y}=[1,0,\cdots,0]^*$ with $\vert\vert{\bf y}\vert\vert _x=1$, by which the greatest eigenvalue $\lambda_{max}$ is maximally weighted while all others are weighted by 0. As also $\vert\vert{\bf x}\vert\vert=\vert\vert{\bf y}\vert\vert=1$, we therefore have
    

    $$\vert\vert{\bf A}\vert\vert _2=\sup_{\vert\vert{\bf x}\vert... ..._2=1}\{\vert\vert{\bf Ax}\vert\vert _2\}=\sqrt{\lambda_{max}}$$
    
    
    • Subordinance

      If vector ${\bf x}$ is the eigenvector corresponding to the greatest eigenvalue $\lambda_{max}$ of ${\bf A}^*{\bf A}$:
      

      $$({\bf A}^*{\bf A}) {\bf x}=\lambda_{max}({\bf A}^*{\bf A})\;{\bf x} =\vert\vert{\bf A}\vert\vert^2\; {\bf x}$$
      
      
      then the equality of the subordinance property $\vert\vert{\bf A}{\bf x}\vert\vert\le\vert\vert{\bf A}\vert\vert \vert\vert{\bf x}\vert\vert$ holds. Consider
      
      $$\vert\vert{\bf A}{\bf x}\vert\vert^2=\langle({\bf Ax})^*, ... ... =\vert\vert{\bf A}\vert\vert^2 \vert\vert{\bf x}\vert\vert^2$$
      
      
      taking square root on both sides we get $\vert\vert{\bf A}{\bf x}\vert\vert=\vert\vert{\bf A}\vert\vert \vert\vert{\bf x}\vert\vert$.

    • Submultiplicativity

      The equality of the submultiplicativity property $\vert\vert{\bf AB}\vert\vert\le\vert\vert{\bf A}\vert\vert\;\vert\vert{\bf B}\vert\vert$ holds if ${\bf A}=c{\bf B}^*$ (${\bf A}$ and ${\bf B}$ are linearly dependent). Consider
      

      $$\vert\vert{\bf B}\vert\vert=\sqrt{\lambda_{max}({\bf B}^*{\... ... B}^*)} =\vert c\vert \sqrt{\lambda_{max}({\bf B}^*{\bf B})}$$
      
      
      and
      
      $$\vert\vert{\bf AB}\vert\vert=\vert c\vert \vert\vert{\bf B... ...^*{\bf B})^2} =\vert c\vert \lambda_{max}({\bf B}^*{\bf B})$$
      
      
      i.e., $\vert\vert{\bf A}{\bf B}\vert\vert=\vert\vert{\bf A}\vert\vert\;\vert\vert{\bf B}\vert\vert$.

    • Unitary invariance

      The spectral norm is the only one out of the three matrix norms that is unitary invariant, i.e., it is conserved or invariant under a unitary transform (such as a rotation) ${\bf B}={\bf R}^*{\bf A}{\bf R}$:
      

      $$\vert\vert{\bf B}\vert\vert _2^2 = \lambda_{max}\left({\bf B}^*{\bf B}\right) =\lambda_{max}\left[\... ...t] =\lambda_{max}\left({\bf R}^*{\bf A}^*{\bf R}{\bf R}^*{\bf A}{\bf R}\right)$$

      $$= \lambda_{max}\left( {\bf R}^*{\bf A}^*{\bf A}{\bf R} \right) =\lambda_{max}\left( {\bf A}^*{\bf A} \right)=\vert\vert{\bf A}\vert\vert _2^2$$

      
      
      Here we have used the fact that the eigenvalues and eigenvectors are invariant under the unitary transform.

  Example

  
  

  $${\bf A}=\left[\begin{array}{rrr}3&-6&2 2&5&1 -3&2&2\end... ...t[\begin{array}{rrr}3&2&1 2&-3&0 1&0&-1\end{array}\right]$$
  
  

  
  

  $$\vert\vert{\bf A}\vert\vert _1 = \max\{\vert 3\vert+\vert 2\vert+\vert-3\vert,\;\vert-6\vert+\vert... ...\vert 2\vert,\;\vert 2\vert+\vert 1\vert+\vert 2\vert\} =\max\{8,\;13,\;5\}=13$$

  $$\vert\vert{\bf A}\vert\vert _\infty = \max\{\vert 3\vert+\vert-6\vert+\vert 2\vert,\;\vert 2\vert+\vert... ...\vert 1\vert,\;\vert-3\vert+\vert 2\vert+\vert 2\vert\} =\max\{11,\;8,\;7\}=11$$

  
  
  The eigenvalues of ${\bf A}^*{\bf A}$ are
  
  $$\lambda_1=69.353,\;\;\;\lambda_2=17.967,\;\;\;\lambda_3=8.680$$
  
  
  The singular values of ${\bf A}$ are
  
  $$\sigma_1=\sqrt{\lambda_1}=8.328,\;\;\;\sigma_2=\sqrt{\lambda_2}=4.239, \;\;\;\sigma_3=\sqrt{\lambda_3}=2.946$$
  
  
  The norm of ${\bf A}$ is
  
  $$\vert\vert{\bf A}\vert\vert _2=\sigma_{max}=\sqrt{\lambda_{max}}=8.328$$
  
  
  The eigenvector corresponding to greatest eigenvalue $\lambda_1$ is ${\bf x}=[0.285,\;-0.957, 0.057]^T$, which satisfies the equality $\vert\vert{\bf Ax}\vert\vert=\vert\vert{\bf A}\vert\vert\;\vert\vert{\bf x}\vert\vert$.

• The Schatten norms

  The Shatten norm is defined based on the singular values $\sigma_i$ of ${\bf A}$ or the eigenvalues $\lambda_i=\sigma^2$ of ${\bf A}^*{\bf A}$:
  

  $$\vert\vert{\bf A}\vert\vert _p=\left(\sum_{i=1}^R \sigma_i^... ...eft(\sum_{i=1}^R \left(\sqrt{\lambda_i}\right)^p\right)^{1/p}$$
  
  
  In particular, consider three common $p$ values:
  • $p=1$ is the nuclear or trace norm:
    
    $$\vert\vert{\bf A}\vert\vert _1=\sum_{i=1}^R \sigma_i=\sum_{... ... \sqrt{\lambda_i} =tr\left( \sqrt{{\bf A}^*{\bf A} } \right)$$
    
    

  • $p=2$ same as the Frobenius norm:
    
    $$\vert\vert{\bf A}\vert\vert _2=\sqrt{\sum_{i=1}^R \sigma_i^... ...=\sqrt{\sum_{i=1}^R \lambda_i}=\vert\vert{\bf A}\vert\vert _F$$
    
    

  • $p=\infty$ same as the spectral norm (the induced 2-norm), the spectral radius of ${\bf A}^*{\bf A}$.
    
    $$\vert\vert{\bf A}\vert\vert _\infty=\max\{\sigma_1,\cdots,\sigma_n\}=\sigma_{max} =\sqrt{\lambda_{max}}$$
    
    

  As the eigenvalues and eigenvectors of ${\bf A}^*{\bf A}$ are invariant under unitary transform, the Schatten norms are unitary invariant as well.

All matrix norms defined above are equivalent according to the theorem previously discussed.

• The Frobenius norm $\vert\vert{\bf A}\vert\vert _F=\sqrt{\sum_{i=1}^R \lambda_i}$ and the induced 2-norm $\vert\vert{\bf A}\vert\vert _2=\sqrt{\lambda_{max}}$ are equivalent:
  
  $$\vert\vert{\bf A}\vert\vert _2\le\vert\vert{\bf A}\vert\vert _F\le\sqrt{R}\;\vert\vert{\bf A}\vert\vert _2$$
  
  
  The equality on the left holds when all eigenvalues $\lambda_i$ but one are zero, and the equality on the right holds when all $\lambda_i$ are the same.
• The Frobenius norm $\vert\vert{\bf A}\vert\vert _F=\sqrt{\sum_{i=1}^R \lambda_i}$ and the Schatten 1-norm $\vert\vert{\bf A}\vert\vert _1=\sum_{i=1}^R \sqrt{\lambda_i}$ are equivalent:
  
  $$\vert\vert{\bf A}\vert\vert _F\le\vert\vert{\bf A}\vert\vert _1\le\sqrt{R}\;\vert\vert{\bf A}\vert\vert _F$$
  
  
  The equality on the left holds when all eigenvalues $\lambda_i$ but one are zero, and the equality on the right holds when all $\lambda_i$ are the same.
• The element-wise maximum norm $\vert\vert{\bf A}\vert\vert _{max}=\max_{i,j}\{\vert a_{ij}\vert\}$ and the Frobenius norm $\vert\vert{\bf A}\vert\vert _F=\sqrt{\sum_{i=1}^M\sum_{j=1}^N\vert a_{ij}\vert^2}$ are equivalent:
  
  $$\vert\vert{\bf A}\vert\vert _{max}\le \vert\vert{\bf A}\vert\vert _F\le \sqrt{MN}\;\vert\vert{\bf A}\vert\vert _{max}$$
  
  
  The equality on the left holds when all elements $\lambda_i$ but one are zero, and the equality on the right holds when all elements are the same.
• 
  
  $$\frac{1}{\sqrt{N}}\vert\vert{\bf A}\vert\vert _\infty\le\ve... ...A}\vert\vert _2\le\sqrt{M}\vert\vert{\bf A}\vert\vert _\infty$$
  
  
  Proof: Define an N-D vector ${\bf e}=[1,\cdots,1]^T$, then the greatest absolute row sum of ${\bf A}$ is
  
  $$\vert\vert{\bf A}\vert\vert _\infty=\max_i\sum_{j=1}^N\vert... ...rt{\bf e}\vert\vert _2=\sqrt{N}\vert\vert{\bf A}\vert\vert _2$$
  
  
  i.e.,
  
  $$\frac{1}{\sqrt{N}}\vert\vert{\bf A}\vert\vert _\infty\le\vert\vert{\bf A}\vert\vert _2$$
  
  

• 
  
  $$\frac{1}{\sqrt{M}}\vert\vert{\bf A}\vert\vert _1\le\vert\vert{\bf A}\vert\vert _2\le\sqrt{N}\vert\vert{\bf A}\vert\vert _1$$
  
  

Theorem

$$\rho({\bf A})\le \vert\vert{\bf A}^k\vert\vert^{1/k}$$

Proof: Let $\lambda$ and ${\bf\phi}$ by the eigenvalue and the corresponding eigenvector of ${\bf A}$ respectively, i.e.,

$${\bf A}{\bf\phi}=\lambda{\bf\phi},\;\;\;\;\mbox{and}\;\;\;\;\; {\bf A}^k{\bf\phi}=\lambda^k{\bf\phi}$$

Taking norm on both sides we get

$$\vert\vert\lambda^k{\bf\phi}\vert\vert=\vert\lambda\vert^k\v... ...le\vert\vert{\bf A}^k\vert\vert\;\vert\vert{\bf\phi}\vert\vert$$

Dividing both sides by $\vert\vert{\bf\phi}\vert\vert\ne 0$ we get

$$\lambda^k\le \vert\vert{\bf A}^k\vert\vert,\;\;\;\;\;\mbox{i.e.}\;\;\;\;\; \lambda\le \vert\vert{\bf A}^k\vert\vert^{1/k}$$

Theorem A square matrix ${\bf A}$ is convergent, i.e., $\lim_{n\rightarrow\infty}{\bf A}^n=0$, if and only if $\rho({\bf A})<1$.

The proof of this theorem can be found here.