Two-Sample t-Test

Given two sets of samples $\{x_1,\cdots,x_{N_x}\}\sim {\cal N}(\mu_x,\sigma^2_x)$ and $\{y_1,\cdots,y_{N_y} \}\sim {\cal N}(\mu_y,\sigma^2_y)$ drawn from their respective normal distributions, we can carry out either Z-test or t-test to either accept or reject the null hypothesis $H_0$: the two populations have the same mean i.e., $\mu_x=\mu_y$.

The two-sample t-test is widely used to compare two populations and see if they are significantly different from each other. Examples include comparison of public and private schools in test scores, American and Japanese cars in fuel efficiency, treatment (drug) and control (placebo) groups in effectiveness of a drug, etc.

We first find the sample means

  $$\bar{x}=\frac{1}{N_x}\sum_{i=1}^{N_x} x_i\sim{\cal N}\left(\mu_x,... ...{N_y}\sum_{i=1}^{N_y} y_i\sim{\cal N}\left(\mu_y,\frac{\sigma^2_y}{N_y}\right)$$ (37)

Consider

  $$\bar{x}-\bar{y} \sim {\cal N}\left(\mu_x-\mu_y,\left(\frac{\sigma_x^2}{N_x}+\frac{\sigma_y^2}{N_y}\right)\right)$$ (38)

which as an algebraic sum of two normally distributed variables is also normally distributed. The test statistic below has a standard normal distribution under the null hypothesis $H_0:\;\mu_x=\mu_y$:

  $$z=\frac{\bar{x}-\bar{y}}{\sqrt{\frac{\sigma_x^2}{N_x}+\frac{\sigma_y^2}{N_y}}} \;\sim \;{\cal N}(0,1)$$ (39)

Given a significance level $\alpha$, we can find the critical value $z_{\alpha/2}$ from the z-Table, and the null hypothesis $H_0$ is rejected if $\vert z^*\vert>\vert z_{\alpha/2}\vert$.

If $\sigma^2_x$ and $\sigma^2_y$ are unknown, we need to estimate them by the sample variances:

  $$S^2_x=\frac{1}{N_x-1}\sum_{i=1}^{N_x} (x_i-\bar{x})^2,\;\;\;\;\;\;\;\;\;\;\;\; S^2_y=\frac{1}{N_y-1}\sum_{i=1}^{N_y} (y_i-\bar{y})^2$$ (40)

and construct a test statistic

  $$t=\frac{\bar{x}-\bar{y}}{\sqrt{\frac{S_x^2}{N_x}+\frac{S_y^2}{N_y}}}$$ (41)

This is called Welch's t-test, where the pdf of test statistic $t$ is approximately a t-distribution with the following degrees of freedom (Welch-Satterthwaite equation):

  $$\nu=\frac{(S_x^2/N_x+S_y^2/N_y)^2}{(S_x^2/N_x)^2/(N_x-1)+(S_y^2/N_y)^2/(N_y-1)}$$ (42)

If the sample variances $S_x^2$ and $S_y^2$ are similar to each other, we can assume that the two data sets have the same variance, and find the pooled variance of $\bar{x}-\bar{y}$ to approximate both $S_x^2$ and $S_y^2$:

  $$S^2_p=\frac{1}{N_x+N_y-2}\left(\sum_{i=1}^{N_x}(x_i-\bar{x})^2 +\... ...i-\bar{y})^2\right) =\frac{1}{N_x+N_y-2}\left((N_x-1)S_x^2+(N_y-1)S_y^2\right)$$ (43)

and test statistic becomes

  $$t=\frac{\bar{x}-\bar{y}}{\sqrt{\frac{S_x^2}{N_x}+\frac{S_y^2}{N_y... ...\bar{y}}{S_p\sqrt{\frac{1}{N_x}+\frac{1}{N_y}}} \;\sim\; {\cal T}_{N_x+N_y-2}$$ (44)

with a t-distribution of $N_x+N_y-2$ degrees of freedom. Substituting the calculated $\bar{x}$, $\bar{y}$ and ESE into this expression of $t$, we get the specific value $t^*$, based on which we can further fine the p-value from the t-Table:

  $$p=P(\vert t\vert>\vert t^*\vert)=1-P(t<\vert t^*\vert)=1-\int_{-\vert t^*\vert}^{\vert t^*\vert} {\cal T}_{\nu}(\tau)\;d\tau$$ (45)

If this p-value is smaller than a pre-determined significance level $\alpha$, we conclude that the data provide significant evidence to reject the null hypothesis $H_0:\;\mu_x=\mu_y$.

We can also find the lower and upper limits of the confidence interval of the test statistic $\bar{x}-\bar{y}$ given a significant $\alpha$:

  $$L=(\bar{x}-\bar{y})\;+t_{\alpha/2} S_p\sqrt{1/N_x+1/N_y} \;\;\;\;\;\;\; U=(\bar{x}-\bar{y})\;-t_{\alpha/2} S_p\sqrt{1/N_x+1/N_y}$$ (46)

so that $P(L\le (\bar{x}-\bar{y})\le U)=1-\alpha$.

The Matlab function ttest2 can be used to carry out two-sample t-tests.

Example: Given two sets of data below and $\alpha=0.05$,

  $$\begin{tabular}{c\vert\vert c\vert c\vert c\vert c\vert c\vert c\... ...hline Data set 2 & 20.2 & 16.9 & 18.5 & 17.3 & 20.5 & & \\ \hline \end{tabular}$$ (47)

We get $m_x=20.91$, $m_y=18.68$, $S_x^2=2.94$, $S_y^2=2.68$

• If we assume the variances of the two data sets are different, then $\nu=9.03$, $t^*=2.27>t_{1-\alpha/2}=2.27$ is in the critical region, and $p=0.048<\alpha=0.05$. The confidence interval of the difference between the two means is $[0.037,\;\;4.4332]$.
• If we assume the variances of the two data sets are the same, then $\nu=10$, $t^*=2.27>t_{1-\alpha/2}=2.23$ is in the critical region, and $p=0.047<\alpha=0.05$. The confidence interval of the difference between the two means is $[0.023,\;\;4.446]$.

In either case, the null hypothesis claiming that the two data sets have the same mean is rejected.