Please enable JavaScript.

Coggle requires JavaScript to display documents.

OpenDT, Variational Inference, Model Interpretability - Coggle Diagram

- - - - Use PyTorch source code along with Bayesian-Torch BaseRNN class to create a variational GRU layer
        
        Understand LSTM and GRU theory/math->Tensors, so many tensors
        
        Understand PyTorch tensors, Autograd, etc.
        
        Benchmark LSTM vs LSTMreparameterization
        
        Both pretty good for solar power
        
        Both pretty bad at load and wind power
        
        LSTMr has some 'step' behavior in predictions, smooths out with more hidden units
        
        Try to produce similar results with custom GRUr
        
        Custom GRUr results, no tuning, one 50% dropout layer
        
        Results are pretty similar! Some kinks to workout for both variational inference versions of LSTM and GRU
        
        Why are recurrent models across the board bad at predicting wind and load power?
        
        inverse_transform back to original scale (acting wonky currently)
        
        Try predicting only load power or wind power, maybe model is prioritizing one output over others?
        
        No significant improvement, dataset simply does not provide enough features that contribute to wind power
        
        AIMS exclusive variational GRU layer 😎
        
        Complicate the model
        
        Stack GRUv layers
        
        Add more linear layers
        
        Built and trained:
        1 linear
        1 GRU
        2 linear
        1 output
        
        1 more item...
        
        Made custom function to count parameters for PyTorch models
        Layer 0 (GRU):
        weight_ih_l0: 384 parameters
        weight_hh_l0: 768 parameters
        bias_ih_l0: 48 parameters
        bias_hh_l0: 48 parameters
        Total parameters in Layer 0: 1248
        Layer 1 (Linear):
        weight: 48 parameters
        bias: 3 parameters
        Total parameters in Layer 1: 51 Total trainable parameters in the model: 1299
        
        1 more item...
        
        Hyperparameter tuning
        
        Tried epochs first, epochs=50 gives 'predicting the mean' behavior :cry:
        
        1 more item...
        
        Near time to start running on GPU, LSTMv takes notably longer than LSTM, especially including the sampling after training to show confidence intervals
        
        Validate LSTMv takes longer than GRUv in more complicated architecture, use 50 epochs
        
        GRUv is faster for both training and inference, but not by a particularly large margin
        
        LSTMr results, no tuning, one 50% dropout layer
        
        Standard LSTM from from nn.LSTM, no tuning
        
        How does ARIMA performance compare? Need to justify use of NNs
        
        Use statsmodels package to implement ARIMA on same ORNL 1-week data