Please enable JavaScript.

Coggle requires JavaScript to display documents.

Megazord, Vanilla : - Coggle Diagram

- - - - Linear Subnetwork Builder
        
        forward_fn
        
        apply_fn
        
        Linear Transformer
        
        OrderedDAG
        
        TransformerBlock x n_layers
        
        Parallel Attention Transformer
        
        MTJ Transformer
        
        VanillaTransformerBlock
        
        Abstract Block
        
        2 more items...
        
        Concrete Block
        
        1 more item...
        
        InputEmbeddingLayer
        
        OutputEmbeddingLayer
        
        block_groups
        
        Group DAG blocks into List[(block, n_repeat)]
        
        init_fn
        
        init model_state
        
        trainable/non-trainable params
        
        {layer_name: {rep_N: params}}
        
        finetuning.n_last_layers
        
        step #
        
        random key
        
        opt_state
        
        last_layer
        
        shared
        
        insert rep_{n} in node name
      - build
        
        train_fn
        
        calc loss & grad (jax.value_and_grad)
        
        break if loss = NaN
        
        update model_state (optax)
        
        gradient accumulation
        
        forward_fn
        
        jax.lax.scan over n_gas
        
        eval_fn
        
        forward_fn
        
        compute metrics
      - Supervised Contrastive Subnetwork Builder
      - Large Contrastive Subnetwork Builder
    - - 8 cores