Please enable JavaScript.

Coggle requires JavaScript to display documents.

Bayesian Statistics - Coggle Diagram

- - - - \(BF > 1\)
        Data provides evidence for the hypothesis
        \(BF < 1\)
        Data provides evidence against the hypothesis
        \(BF = 1\)
        Data provides no evidence
- - - - \(P(a < \theta < b) = \int^b_a 1*f(\theta^\ast)d \theta\)
      - Standard Error
        \(\sqrt \frac{VAR(\theta^\ast)}{m} \)
  - - - Monte Carlo Effective Sample size
        Number of independent samples required from the stationary distribution you would have.
        or number of steps required to have samples that are no longer correlated to each other
  - - - Linear Regression (real input, real output)
        \(y_i = \beta_0 + \beta_1 x_{1i} + ... + \beta_k x_{ki} + \epsilon_i\) where \(\epsilon_i \sim N(0, \sigma^2)\)
        \(y_i|x_i, \beta_i, \sigma^2 \sim N(\beta_0 + \beta_1 x_{1i} + ... + \beta_k x_{ki},\sigma^2)\)
        \(\beta_0 \sim p(\beta_0) ...\)
        \(\sigma^2 \sim p(\sigma^2)\)
      - ANOVA (categorical input, real output)
        \(y_i|g_i,\mu_{gi},\sigma^2 \sim N(\mu_{gi}, \sigma^2)\) or different variance for each mean
        \(E(y_i) = \mu = \beta_0 + \beta_1 x_1 + ... + \beta_{G-1} x_{G-1}\)
        where \(x_i = I(g=1), ..., x_{G-1} = I(g = G-1)\) so x are indicators
      - Logistic Regression (real input, binary output)
        \(y_i | \theta_i \sim Bern(\theta_i)\)
        Link Function, logit
        \(logit(\theta) = log(\frac{\theta}{1-\theta}) = \beta_0 + \beta_1 x_1 + ...\)
        \(E(y_i) = \theta_i = ... = \frac{e^{ \beta_0 + \beta_1 x_1 + ...}}{1 + e^{ \beta_0 + \beta_1 x_1 + ...}} = \frac{1}{1 + e^{-( \beta_0 + \beta_1 x_1 + ...)}}\)
        
        This is Sigmoid Function!
        
        Prediction
        \(y_i = I(\theta_i > \alpha) = I(\frac{1}{1 + e^{-( \beta_0 + \beta_1 x_1 + ...)}} > \alpha)\) \(\alpha\) is threshold
      - Poisson Regression (real input, real count output)
        \(y_i|\lambda_i \sim Pois(\lambda_i)\) and E(y_i) = Var(y_i)
        Link Function, log
        \(log(\lambda_i) = \beta_0 + \beta_1 x_1 + ...\)
        \(E(y_i) = \lambda_i = e^{\beta_0 + \beta_1 x_1 + ...}\)
- - - - Conditional Risk (or Expected Loss)
        Given an observation 'x' and take action \(\alpha_i\), the conditional risk is follow:
        \(R(\alpha_i|x) = \sum_{j=1}^c \lambda(a_i|\omega_j)P(\omega_j | x)\)
      - Cost
        If cost are not equal for classifications, cost can be include in the risk
      - Overall Risk
        \(R^* = \int R(\alpha(x) | x) p(x) dx\)
  - - - Minimum risk rule
        
        Choose action with lower risk
        \(min_{a_i}\{R(a_1|x), R(a_2|x), ..., R(a_l|x)\}\)
        
        \((\lambda_{21} - \lambda_{11}) P(\omega_1 | x) > (\lambda_{12} - \lambda_{22}) P(\omega_2 | x)\)
        
        \(\frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{(\lambda_{12} - \lambda_{22}) P(\omega_2)}{(\lambda_{21} - \lambda_{11}) P(\omega_1)}\)
      - Minimax Criterion
        Find prior \(P(\omega_1)\) that the Bayes risk is maximum
      - Neyman-Pearson Criterion
        Minimize the overall risk subject to a contastraint
        \(\int R(a_i|x) dx < \gamma\)
        
        There is a fixed resources accompanying action \(\alpha_i\)
        
        Must not misclassify more than the specific frequency
    - - Symmetrical (zero-one) Loss
        \(\lambda(\alpha_i | \omega_j) = \) {0 if \(i = j\), 1 if \(j \neq j\)}
        penalizing for making a wrong decision
        
        All errors are equally costly
        
        Risk
        \(R(\alpha_i|x) = \sum_{j=1}^c \lambda(a_i|\omega_j)P(\omega_j | x)\)
        \(= \sum_{j \neq i} P(\omega_j | x)\)
        \(= 1 - P(\omega_i | x)\)
        Note \(\lambda(a_i|\omega_j) = 0\) when \(i = j\)
        
        Adjusting Penalty
        \(\lambda_{ij}\) may be adjusted to output higher loss for making a specific 'mistake'
        \(\lambda_{12} > \lambda_{21}\) gives higher penalty for taking \(\alpha_1\) when the state of nature is \(\omega_2\)
  - - - Dichotomizer
        Two-category only case
        \(g(x) = g_1(x) - g_2(x)\)
        Choose \(\omega_1\) if \(g(x) > 0\)
        Choose \(\omega_2\) otherwise
      - General Case
        Multi-category case
        \(g_i(x) = -R(\alpha_i | x)\)
        Simplifies in minimum-error-rate case:
        \(g_i(x) = P(\omega_i | x)\)
        Higher the \(g_i(x)\), lower the risk #
    - - Case 1 \(\Sigma_i = \sigma^2 I\)
        Minimum distance classifier
        
        Statistically independent
        
        Different mean, same variance
        
        Equaltion simplfies to \(g_i(x)\)
      - Case 2 \(\Sigma_i = \Sigma\)
        
        Each entry in all Sigma is all same
        
        \(Det(\Sigma) = 0\)
        
        Taking Average each entry
        
        Taking Maximum each entry
        
        Taking most/least observation
        
        Taking sigma that has most variability
      - Case 3 \(\Sigma_i = ?\)
        
        Decision boundaries are non-linear