Please enable JavaScript.
Coggle requires JavaScript to display documents.
Week 10 - Statistical Significance (Comparing Distributions (Statistical…
Week 10 - Statistical Significance
Comparing Distributions
Is it the same
underlying distribution?
Summary Statistics:
not enough to draw conclusions
Are the 2 distributions
the same/distinguishable?
Statistical Significance Testing
Use of a test based on some statistics of the distributions to determine whether the difference between the distributions are significant enough that the null hypothesis about the 2 distributions can be rejected, in favour of the alternative hypothesis.
How these tests work
Null Hypothesis
Assumed to be true
Data analysis is performed to
accumulate statistical evidence
to reject H0 in favour of alternative Ha
represents
status quo
:
The different between the
mean height of sons and the
mean height of fathers = 0
Problem
:
is there enought statistical evidence
to reject H0, thus requiring that Ha is true?
H0: mean (sheight - fheight) = 0
Ha: mean (sheight - fheight) > 0
Outcomes of Hypothesis Testing:
p-values
Usually try to decrease
likelihood of type I error
The lower the p-value,
lower likelihood of type I error
Probability of accepting Ha,
when in fact it is not true
Limitation:
Sample Size
the actual subset of data,
may have influenced outcome
size of data
influences p-value
If smaller sample,
subset of original data,
get different result
limited amount of data
type II error:
might accept H0 when
Ha was actually true
not
necessarily
going to reject H0 with
increased data though
test under-powered:
not enough evidence to reject H0
Interpret results
of significance tests
alpha & beta
alpha
the acceptable
amount of
type 1 errors
the likelihood of
rejecting H0 when
instead it was true
if p = 0.05:
1 out of 20 times
when we reject H0,
we are wrong
beta
prob of type II error
fail to reject H0
when it is false
incorrectly retaining H0,
thus discarding Ha
power
prob of correctly rejecting H0
1 - beta
paired test
effects the computation
of the thresholds for
determining significance
paired versus unpaired
parameter that
describes the conditions
under which the test is run
t-test
Determine prob of making
type I error if we reject H0
alpha
acceptable prob for
type I error to occur
limit on the acceptable
p-value to reject H0
acceptable prob for
rejecting H0 while it is true
usually 0.05 or 0.01
if p < 0.05 (ie p < alpha),
reject H0, in favour of Ha
allows us to
compute p-values
Assumes data
has t-distribution
p-hacking
less commonly found
when clear H testing
more likely when analysing
a dataset to understand if
observations measured across
2 groups are statistically significant
use of data mining & machine learning
to uncover patterns in data
that reveal statistical significance,
without first devising a
specific hypothesis to test
avoiding
occurs if measure
statistical tests on the
same data we use to
discover such patterns
instead
:
split data into 2 sets
discover patterns
test whether H0
we have from patterns
are found in the unseen data
& if the patterns are
statistically significant
