Hypothesis Testing 101

One-tailed and two-tailed tests (Example)

Statistical significance (false positive and type I error)

Statistical power

Calculation

Power analysis

P-value

Confidence interval

Parametric and nonparametric tests

Parametric tests (based on certain assumptions)

Nonparametric tests (Distribution-free tests based on fewer assumptions)

Margin of error = half of width of CI = critical value * standard error of the statistic

Z test

T-test

mean or proportion of one group to a reference value

means or proportions of two groups

mean or proportion of one group to a reference value

means or proportions of two groups

ANOVA

means of multiple groups, All groups have the same means

F-test

means of multiple groups, proposed regression model fits the data

Cost is less powerful

Chi-square test

Fisher's exact test

Permutation tests(Re-randomization tests)

Conditional probability

It measures the probability of getting the observed data or more extreme data given that the null hypothesis is true.

A very small p-value means that the probability of observing the data due to chance is very low. A p-value < 0.05 denotes strong evidence against the null hypothesis, which means the null hypothesis can be rejected.

A p-value > 0.05 denotes weak evidence against the null hypothesis, which means the null hypothesis cannot be rejected.

A p-value = 0.05 is a marginal value indicating it is possible for the test to go either way; in practice, the null hypothesis is usually not rejected.

Computer the rejection region and critical value (Z)

Beta, Power=1 - P(Type II error)

Type I error (False positive)

Type II error (False negative)