Derivative Approximated by Finite Difference

The derivative of a given function $f(x)$ can be approximated by the forward difference based on a step size $h$:

$$D[f]=\frac{d}{dx}f(x)=f'(x)=\lim\limits_{h\rightarrow 0}\frac{f(x+h)-f(x)}{h} \approx \frac{f(x+h)-f(x)}{h}$$ (1)

The same result can be obtained based on the Taylor expansion of $f(x)$:

$$f(x+h)=f(x)+hf'(x)+\frac{h^2}{2}f''(x)+\frac{h^3}{3!}f'''(x) +\frac{h^4}{4!}f^{(4)}(x)+\frac{h^5}{5!}f^{(5)}(x)+\cdots$$ (2)

Solving for $f'(x)$ we get

$$f'(x)=\frac{f(x+h)-f(x)}{h}-\frac{h}{2}f''(\xi)\approx\frac{f(x+h)-f(x)}{h}, \;\;\;\;\; \xi\in[x,\;x+h]$$ (3)

with a first order truncation error $O(h)=hf''(\xi)/2$ linearly related to $h$ (mean-value form of the remainder. The approximation is poor unless $h$ is sufficiently small.

The derivative of $f(x)$ can also be approximated by the backward difference. Replacing $h$ by $-h$ in the Taylor expansion above, we get

$$f(x-h)=f(x)-hf'(x)+\frac{h^2}{2}f''(x)-\frac{h^3}{3!}f'''(x) +\frac{h^4}{4!}f^{(4)}(x)-\frac{h^5}{5!}f^{(5)}(x)+\cdots$$ (4)

Again solving for $f'(x)$ we can get

$$f'(x)=\frac{f(x)-f(x-h)}{h}+\frac{h}{2}f''(\xi)\approx\frac{f(x)-f(x-h)}{h}$$ (5)

This is the backward difference with the same first order truncation error as the forward difference.

We can also find the central difference by subtracting Eq. (4) from Eq. (2), and dividing both sides by $2h$ to get

$$f'(x) = \frac{f(x+h)-f(x-h)}{2h} -\left(\frac{f'''(x)}{3!}h^2+\frac{f^{(5)}(x)}{5!}h^4+\cdots\right)$$

$$\approx \frac{f(x+h)-f(x-h)}{2h}$$ (6)

with a second order truncation error $O(h^2)$, one order higher than the error of either the forward or backward difference.

Moreover, if we add Eqs. (2) and (4) and solve the resulting equation for $f''(x)$, we get an approximation of the second order derivative:

$$f''(x) = \frac{f(x-h)-2f(x)+f(x+h)}{h^2} -\left(\frac{2f^{(4)}}{4!}h^2+\frac{2f^{(6)}}{6!}h^4+\cdots\right)$$

$$\approx \frac{f(x-h)-2f(x)+f(x+h)}{h^2}$$ (7)

with a second order truncation error $O(h^2)$.