Appendix B: Matrix Operations and Their Derivatives

Assume $A$ and $B$ are matrices.

• $tr(A+B)=tr(A)+tr(B)$, $tr(AB)=tr(BA)$. In the special case when $A$ and $B$ are row and column vectors respectively, $tr(AB)=tr(BA)$ where $AB$ is a scalar and $BA$ is a matrix.
• Assume $X$ is a column vector, then
  
  $$\frac{d}{dX} (X^T A X)=(A+A^T) X$$
  
  
  In particular, when $A^T=A$ is symmetric, we have
  
  $$\frac{d}{dX} (X^T A X)=2A X$$
  
  

• 
  
  $$\frac{d}{dA} \vert A\vert=\left\{ \begin{array}{ll} {\cal A}_{ij} & i=j \\ 2{\cal A}_{ij} & i\ne j \end{array} \right.$$
  
  

• 
  
  $$\frac{d}{dA} log\vert A\vert=2A^{-1}-diag(A^{-1})$$
  
  

• 
  
  $$\frac{d}{dA} tr(AB)=B+B^T-diag(B)$$