Statistic Hypothesis Tests

• Statistical Hypothesis Tests

  Based on an observed data set, a statistical hypothesis test is to find out whether a certain hypothesis should be either accepted or rejected. Typically this is a null hypothesis, denoted by $H_0$, e.g., there is no difference between the mean $\mu$ of the data set and a hypothesized mean $\mu_0$, or between the means $\mu_x$ and $\mu_y$ of two data sets, against an alternative hypothesis, denoted by $H_a$, e.g., the two means are not the same, or one is larger or smaller than the other.

• Test Statistic

  The test of the hypothesis is based on a constructed random variable, the test statistic, with certain probability density function (pdf). For example:

  • z-test:, if the test statistic has a standard normal distribution: $z\;\sim\;{\cal N}(z,0,1)$;
  • t-test: if the test statistic has a Student's t-distribution of degree of freedom (d.f.) $\nu$: $t\;\sim\;{\cal T}_\nu(t)$;
  • f-test: if the test statistic has an F-distribution of degrees of freedom $d_1$ and $d_2$: $f\;\sim\;{\cal F}_{d_1,d_2}(f)$

• Statistic Significance

  The significance of the test result is measured by a pre-determined significance level, denoted by $\alpha$ (typically $\alpha=0.05$), based on which the critical (rejection) region is defined as the region in the domain of the pdf of the test statistic so that the probability for the value of the test statistic to fall inside the region is $\alpha$. The lower bound of the region is the critical value.

• p-value

  The p-value, denoted by $p$, is the probability of finding the observed or more extreme data toward the direction of the alternative hypothesis $H_a$, given that $H_0$ is true.

• Rejection or Acceptance of $H_0$

  If the value of the test statistic falls inside the critical (rejection) region, or, alternatively and equivalently, $p<\alpha$, then the null hypothesis $H_0$ is rejected due to the occurrence of a small-probability event according to $H_0$. Otherwise we fail to reject $H_0$.

  • If $p<\alpha$, we reject $H_0$, and say such a test result is statistically significant, i.e., there is a significant evidence against $H_0$, in favor of $H_a$.
  • If $p>\alpha$, we fail to reject $H_0$, as the evidence against $H_0$ is not significant enough. However, this does not necessarily mean the evidence for $H_0$ is strong.