Vector and matrix differentiation

A vector differentiation operator is defined as

$$\frac{d}{d{\bf x}}\stackrel{\triangle}{=} \left[ \frac{\part... ...\partial x_1},\cdots, \frac{\partial}{\partial x_n} \right]^T$$

which can be applied to any scalar function $f({\bf x})$ to find its derivative with respect to ${\bf x}$:

$$\frac{d}{d{\bf x}} f({\bf x}) = \left[ \frac{\partial f}{\partial x_1},\cdots, \frac{\partial f}{\partial x_n} \right]^T$$

Vector differentiation has the following properties:

$$\frac{d}{d{\bf x}}({\bf b}^T{\bf x})=\frac{d}{d{\bf x}}({\bf x}^T{\bf b})={\bf b}$$

$$\frac{d}{d{\bf x}}({\bf x}^T{\bf x})=2{\bf x}$$

$$\frac{d}{d{\bf x}}({\bf x}^T{\bf A}{\bf x})=({\bf A}^T+{\bf A}){\bf x}$$

To prove the third one, consider the $k$th element of the vector:

$$\frac{\partial}{\partial x_k} ({\bf x}^{T}{\bf A}{\bf x}) ... ...k}x_i+\sum_{j=1}^n a_{kj}x_j \;\;\;\;\;\;\;\;(k=1, \cdots, n)$$

Putting all $n$ elements in vector form, we have the above. If ${\bf A}^T={\bf A}$ is symmetric, then we have

$$\frac{d}{d{\bf x}}({\bf x}^T{\bf A}{\bf x})=2{\bf A}{\bf x}$$

In particular, when ${\bf A}={\bf I}$, we have

$$\frac{d}{d{\bf x}}({\bf x}^T{\bf x})=2{\bf x}$$

You can compare these results with the familiar derivatives in the scalar case:

$$\frac{d}{dx}(ax^2)=2ax$$

A matrix differentiation operator is defined as

$$\frac{d}{d{\bf A}}\stackrel{\triangle}{=}\left[ \begin{array}... ... & ... & \frac{\partial}{\partial a_{mn}} \end{array} \right]$$

which can be applied to any scalar function $f({\bf A})$:

$$\frac{d}{d{\bf A}}f({\bf A})=\left[ \begin{array}{ccc} \frac... ...frac{\partial f({\bf A})}{\partial a_{mn}} \end{array} \right]$$

Specifically, consider $f({\bf A})={\bf u}^T {\bf A} {\bf v}$, where ${\bf u}$ and ${\bf v}$ are $m\times 1$ and $n\times 1$ constant vectors, respectively, and ${\bf A}$ is an $m\times n$ matrix. Then we have:

$$\frac{d}{d{\bf A}} ({\bf u}^T {\bf A} {\bf v}) = {\bf u} {\bf v}^T$$