Please enable JavaScript.

Coggle requires JavaScript to display documents.

Statistical Learning in Practice - Coggle Diagram

- - - - d=1
        
        \(Z\sim P_{\theta,\phi}\)
        \(\mathbb{E}Z=K'(\theta)\)
        (invertible function; identifiable)
        \(Var Z = \phi K''(\theta)\)
        
        Reparametrise to
        \(\mathcal{P}_{\theta,\phi} \to \mathcal{P}_{\mu,\phi}\)
        \(\mu(\theta)=K'(\theta)\)
        \(Var Z = \phi V(\mu)\)
        
        Examples
        
        Normal Distribution wrt Lebesgue measure
        \( \{ N(\gamma, \sigma^2): \gamma \in \mathbb{R}, \sigma^2 >0 \} \)
        with \( K(\theta)=\theta^2/2,\ \mu=\theta=\gamma,\ V(\mu)=1\)
        
        Poisson Distribution wrt counting measure
        \(\{ Pois(\lambda):\lambda>0\}\)
        with \(\theta = \log \lambda,\ \phi=1\)
        \(K(\theta)=e^\theta, \mu=e^\theta=\lambda,\ V(\mu)=\mu\)
        
        Binomial Distribution wrt counting measure
        \(\left\{ \frac{1}{m}Bin(m,p): m\in\mathbb{N}, p\in(0,1) \right\}\)
        with \(\theta = logit(p) = \log \frac{p}{1-p}, \phi = \frac{1}{m} \)
        \( K(\theta) = \log(e^\theta + 1),\ \mu=expit(\theta)=\frac{e^\theta}{1+e^\theta} = p,\ V(\mu)=\mu(1-\mu) \)
      - Family of distributions \(\mathcal{P}\) with
        
        natural param space \(\Theta\)
        
        dispersion param space \(\Phi\)
        
        density \(f\) has form
        \(f(z;\theta,\phi)=h(z,\phi)\exp \left( \frac{1}{\phi}(\theta^T z - K(\theta)) \right) \)
    - - link function (known)
      - dispersion weights (possibly unknown)
      - Definition
    - - MLE \(\hat{\beta}\) exists, \(\hat{\mu}_i\) satisfies a set of score equations
      - \(\hat{\beta}\) can be approximated by Newton-Raphson, Fisher Scoring (IRLS algorithm)
    - - Deviance of GLM \(D(Y;\hat{\mu})\)
        
        Connection with Likelihood Ratio Test Statistic of current model vs saturated model
      - Large sample asymptotics
      - Small dispersion asymptotics
        
        Note Poisson model has fixed dispersion, so does not apply here. But we call Poisson model large count asymptotic result as this
    - - Prediction/Generalisation/Test Error
        
        \(\mathbb{E}(Y^*-\hat{f}(x))^2= \sigma^2 + \Bias^2(\hat{f}(x^*))+\Var(\hat{f}(x^*))\)
      - Information Criteria
        
        AIC
        
        chooses more complex models
        => predictive rather than explanatory
        
        BIC
        
        Penalises dimension more
        => chooses simpler models
      - v-fold Cross Validation
        
        roughly v subsets of equal size
        
        use V-1 subsets to estimate model parameters
        
        use estimated model to test on remaining subset
        
        repeat for to test each subset
        
        report average validation error
      - Best subset, forward, backward selection
        
        see practical 2
      - Regularisation
        
        Minimise an objective function of data and model
        
        data fidelity term
        
        penalisation/regularisation term
        
        Applied to GLM
        
        \(+\lambda \sum_{j=1}^p |\beta_j|^q\)
        with smoothing parameter \(\lambda\)
        
        q=0; best subset selection using
        
        AIC: \(\lambda = 2\)
        
        BIC: \(\lambda = \log n\)
        
        q=1; Lasso
        
        q=2; Ridge
        
        Principle Component Analysis (PCA)
        
        Comparison of OLS, Ridge, PCA, Lasso
  - - - Overdispersion in Poisson regression model
      - \(\{ NB(r,p):r\in\mathbb{N}, p\in(0,1)\}\), \(r\) known
        with \(\mu=r\frac{p}{1-p},\ \phi=1,\ V(\mu)=\mu+\frac{\mu^2}{r}\)
        
        Generalised reprametrisation: \(\tau=r^{-1}\)
        => unknown \(\tau\) means model is not a GLM
        
        Joint estimation of \(\beta,r\) is required
    - - Mixture model \(Y_i = p_i\delta_0 + (1-p_i)\mathcal{M}_i\)
      - ZIGLM:
        
        \(\mathcal{M}_i\) are GLMs
        
        \( h(p_i) = x_i^T\gamma \), unknown \(\gamma\)
      - Expectation Maximisation (EM) algorithm
    - - \( (x_{ij}, Y_{ij}, z_{ij}) \)
        \(1\leq i \leq I,\ 1\leq j \leq J_i\)
        \(n=\sum J_i\)
        
        rank restrictions, p, q
      - \( Y_{ij} | u_i \sim^\text{ind.} P_{\mu_{ij}, \phi_{ij} }\)
        \( g(\mu_{ij}) = x_{ij}^T \beta + z_{ij}^T u_i\)
        
        fixed effects \(\beta\)
        
        random effects \(u_i \sim P(a) \), \(\alpha\) unknown
      - Examples
        
        normal linear mixed model
        
        logistic- and Poisson-normal models
        
        random intercept model
        
        Linear mixed effect model
        
        \(Y=X\beta + \tilde{Z}u + \epsilon \)
        
        \(Y|u \sim N(X\beta + \tilde{Z}u, \sigma^2 I_n ),\ u_i \sim N(0,\Sigma) \)
        
        Estimation: MLE and REML
      - Estimation by numerical computations
      - Prediction using Bayes'
        resulting a maximum a posteriori (MAP) estimator \(\tilde{u}_i\)
      - Parametric bootstrap
    - - Distribution agnostic, mean-variance relationships:
        \(\mathbb{E}Y_i = \mu_i = g^{-1}(x_i^T \beta ) \)
        \(Var(Y_i)= \nu_i(\mu_i)\)
      - Quasi-score function:
        \(u(\beta;Y) = \sum x_i \frac{Y_i-\mu_i(\beta)}{\nu_i(\mu_i(\beta))} \mu_i'(\beta)\)