Common Probability Density functions in Statistical Tests

The following probability density functions (pdf) are commonly used in statistical hypothesis tests:

• Normal (Gaussian) distribution:
    $${\cal N}(x,\mu,\sigma^2) =\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$ (1)
  if $x$ has a normal distribution denoted by $x\;\sim\;{\cal N}(x,\mu,\sigma^2)$, then $ax+b\;\sim\;{\cal N}(a\mu+b,(a\sigma)^2)$, and
    $$z=\frac{x-\mu}{\sigma}\;\sim\;{\cal N}(x,0,1) =\frac{1}{\sqrt{2\pi}}\;e^{-\frac{x^2}{2}}$$ (2)
  has the standard normal distribution with $\mu=0$ and $\sigma=1$.

  

  For example, if $\{x_1,\cdots,x_N\}$ are independent and identically distributed (i.i.d.) samples of $x\sim{\cal N}(\mu,\sigma^2)$, then

    $$\bar{x}=\frac{1}{N}\sum_{n=1}^N x_n\;\sim\;{\cal N}(\mu,\sigma^2/... ...;\;\;\;\;\;\;\;\; \frac{\bar{x}-\mu}{\sqrt{\sigma^2/N}}\;\sim\;{\cal N}(0,1)$$ (3)

• Chi-squared $\chi^2$-distribution:
    $$\chi^2_k(x)=\frac{1}{2^{k/2}\Gamma(k/2)} e^{k/2-1}e^{-x/2}$$ (4)
  where $k$ is the degrees of freedom, and $\Gamma(n)=(n-1)!$ is the gamma function.

  For example, if $\{z_1,\cdots,z_K\}$ are i.i.d. samples of $z\sim{\cal N}(0,1)$, then

    $$x=\sum_{i=1}^k z_i^2\;\sim\;\chi^2_k(x)$$ (5)
  If all $x_k\;(k=1,\cdots,K)$ have the normal distribution with the same variance $x_k\;\sim\;{\cal N}(0,\sigma^2)$, i.e., $x_k/\sigma \;\sim\;{\cal N}(0,1)$, then
    $$\sum_{i=1}^k x_i^2/\sigma^2\;\sim\;\chi^2_k(x), \;\;\;\;\;\mbox{i.e.}\;\;\;\;\;\; \sum_{i=1}^k x_i^2\;\sim\;\sigma^2\chi^2_k(x)$$ (6)
  

• Student's t-distribution (William Sealy Gosset):
    $${\cal T}_{\nu}(t) =\frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\sq... ...Gamma\left(\frac{\nu}{2}\right)} \left(1+\frac{t^2}{\nu}\right)^{-(\nu+1)/2}$$ (7)
  Specially
    $${\cal T}_1(t)=\frac{1}{\pi(1+t^2)},\;\;\; {\cal T}_2(t)=\frac{1}... ...t{3}}{\pi(3+t^2)^2},\;\;\; {\cal T}_\infty(t)=\frac{1}{\sqrt{2\pi}}e^{t^2/2}$$ (8)
  If $z\;\sim\;{\cal N}(0,1)$ and $x\;\sim\;\chi^2_n$, then
    $$t=\frac{z}{\sqrt{x/n}}\;\sim\;{\cal T}_n(t)$$ (9)
  For example, if $\{x_1,\cdots,x_N\}$ are i.i.d. samples of $x\sim{\cal N}(\mu,\sigma^2)$, and $\bar{x}=\sum_{n=1}^N x_n/N$, then
    $$S^2=\frac{1}{N-1}\sum_{n=1}^N (x_n-\bar{x})^2$$ (10)
  has a Chi-squared distribution with $N-1$ degrees of freedom and
    $$t=\frac{x-\bar{x}}{S/\sqrt{N}}\;\sim\;{\cal T}_{N-1}(t)$$ (11)
  has a t-distribution with with $N-1$ degrees of freedom.

  

  The t-distribution approaches the normal distribution when $\nu\rightarrow\infty$. In general, if $N>30$, the t-distribution is close enough to the standard normal distribution.

• F-distribution (after Ronald Fisher):
    $${\cal F}(x,d_1,d_2) =\frac{\sqrt{\frac{(d_1x)^{d_1}d_2^{d_2}}{(d_1 x+d_2)^{d_1+d_2}}}} {x\,B\left(\frac{d_1}{2},\frac{d_2}{2}\right)}$$ (12)
  where $d_1,d_2>0$ are degrees of freedom, and $B(x,y)=\int_0^1 t^{x-1}(1-t)^{y-1}\;dt$ is Beta function. If $x_1$ and $x_2$ are independent and have Chi-squared distribution:
    $$x_n\;\sim\; \chi^2_{d_1}(x_n),\;\;\;\;\;\;\;\;\;\;\;\; x_d\;\sim\; \chi^2_{d_2}(x_d)$$ (13)
  then their ratio has an F-distribution:
    $$\frac{x_n/d_1}{x_d/d_2}\;\sim\;{\cal F}(x,d_1,d_2)$$ (14)
  For example, if
    $$x_n=\sum_{i=1}^{d_1} u_i^2,\;\;\;\;\;\;\; x_d=\sum_{i=1}^{d_2} v_i^2,\;\;\;\;\;\;\;u_i,v_i\;\sim\;{\cal N}(0,\sigma^2)$$ (15)
  then
    $$x_n\;\sim\;\sigma^2\chi^2_{d_1}(x_n),\;\;\;\;\;\; x_d\;\sim\;\sigma^2\chi^2_{d_2}(x_d)$$ (16)
  and we still have
    $$\frac{x_n/d_1}{x_d/d_2}\;\sim\;{\cal F}(x,d_1,d_2)$$ (17)

  

Assume $x\sim{\cal N}(\mu,\sigma^2)$ is a random variable having a normal probability density function (pdf) with mean $\mu$ and variance $\sigma^2$, both of which are unknown. We can estimate the value of $x$ by the sample mean of a set of $N$ i.i.d samples $\{x_1,\cdots,x_N\}$ drawn from this distribution:

  $$\bar{x}=\frac{1}{N}\sum_{i=1}^N x_i \;\;\sim\;\; {\cal N}(\mu, \sigma^2/N)$$ (18)

This is also a normally distributed random variable with the same mean $\mu$ as $x$ but a different variance $\sigma^2/N$. The standard deviation $\sigma/\sqrt{N}$, called the standard error and denoted by SE, can be considered as the variability or noise. Specially when $N=1$, SE is the same as the standard deviation $\sigma$ of the original distribution, but it decrease when $N$ increases, and it approaches to zero when $N$ approaches to infinity.

We can further define another random variable $z$ by shifting $\bar{x}$ by its mean $\mu$ and scaling by its standard error $SE=\sigma/\sqrt{N}$, so that $z$ has a standard normal distribution of zero mean and unit variance:

  $$z=\frac{\bar{x}-\mu}{\mbox{SE}}=\frac{\bar{x}-\mu}{\sigma/\sqrt{N}} \;\;\sim\;\; {\cal N}(0,1)$$ (19)

Given $z$, called test statistic, we can carry out various statistic tests called Z-test.

If the variance $\sigma^2$ is unknown, it can be estimated by the sample variance:

  $$S^2=\frac{1}{N-1}\sum_{i=1}^N e_i^2 =\frac{1}{N-1}\sum_{i=1}^N (x_i-\bar{x})^2 =\frac{1}{N-1}\left(\sum_{i=1}^N x_i-N\bar{x}\right)$$ (20)

This is an unbiased estimation of the variance, based on which we can further find the unbiased estimated standard error (ESE) $\mbox{ESE}=S/\sqrt{N}$. Now the random variable $z$ based on $\sigma$ can be replaced by another random variable $t$ based on $S$:

  $$t=\frac{\bar{x}-\mu}{\mbox{ESE}}=\frac{\bar{x}-\mu}{S/\sqrt{N}} \sim {\cal T}_\nu(t)$$ (21)

Different from $z$ with a normal distribution, $t$, as a test statistic, has a t-distribution with $\nu=N-1$ degrees of freedom.

Given $t$ as a test statistic, we can carry out various statistic tests called Student's t-test.