Please enable JavaScript.

Coggle requires JavaScript to display documents.

LLM Systems Seminar (CS7670): Week12a, Training in Practice (paper, HotCRP…

- - - - Q: What problem does TorchTitan solve?
- - - - Fully Sharded Data Parallel
    - - Tensor Parallel
    - - Pipeline Parallel
  - - - A DTensor is defined by:
        
        Global shape: shape of the logical tensor.
        
        Device mesh: logical arrangement of devices (e.g., 1D or 2D mesh).
        
        Placements: how each tensor dimension is placed across the mesh:
        
        Replicate() — each device holds a full copy.
        
        Shard(dim) — shard tensor along dim across mesh dimension.
        
        Partial() — each device holds a partial result that must be reduced.
      - DTensor examples
    - - Q: FSDP1 vs. FSDP2?
        
        FSDP1 -> FSDP2
    - - RingAttention
      - Ulysses Sequence Parallelism
        
        DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
        
        Low-Latency and High-Throughput Inference for Long Context with Sequence Parallelism (aka Arctic Ulysses)
  - - - PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation (ASPLOS'24)
      - kernel fusion
        
        1) find all fusion opportunities;
        
        2) score each of the fusion opportunities and sort by that score;
        
        1) the category of the fusion (e.g. pointwise/reduction/template);
        
        2) estimated bytes of memory traffic saved by the fusion;
        
        and 3) shorter distance between nodes in the original graph
        
        3) for each fusion opportunity, check if that fusion remains legal and if so apply it
      - kernel compilation
    - - Reducing Activation Recomputation in Large Transformer Models
    - - Accelerating PyTorch with CUDA Graphs
- - - - H100 95GB x8
      - NVSwitch
      - two hosts on a rack with a TOR
    - - Q: Why does torch.compile deliver larger gains at 128 GPUs than at 8 GPUs?
    - - Q: What does the configuration represent, and how are machines grouped under this setup?
    - - Q: Which config is better?
        
        Q: If we had a technique that cuts memory usage by half at the cost of a 10% throughput drop, would you apply it?