Hypothesis Testing 101
One-tailed and two-tailed tests (Example)
Statistical significance (false positive and type I error)
Statistical power
Calculation
Power analysis
P-value
Confidence interval
Parametric and nonparametric tests
Parametric tests (based on certain assumptions)
Nonparametric tests (Distribution-free tests based on fewer assumptions)
Margin of error = half of width of CI = critical value * standard error of the statistic
Z test
T-test
mean or proportion of one group to a reference value
means or proportions of two groups
mean or proportion of one group to a reference value
means or proportions of two groups
ANOVA
means of multiple groups, All groups have the same means
F-test
means of multiple groups, proposed regression model fits the data
Cost is less powerful
Chi-square test
Fisher's exact test
Permutation tests(Re-randomization tests)
Conditional probability
It measures the probability of getting the observed data or more extreme data given that the null hypothesis is true.
A very small p-value means that the probability of observing the data due to chance is very low. A p-value < 0.05 denotes strong evidence against the null hypothesis, which means the null hypothesis can be rejected.
A p-value > 0.05 denotes weak evidence against the null hypothesis, which means the null hypothesis cannot be rejected.
A p-value = 0.05 is a marginal value indicating it is possible for the test to go either way; in practice, the null hypothesis is usually not rejected.
Computer the rejection region and critical value (Z)
Beta, Power=1 - P(Type II error)
Type I error (False positive)
Type II error (False negative)