Please enable JavaScript.

Coggle requires JavaScript to display documents.

LLM Systems Seminar (CS7670): Week06a, Training Straggler (paper, HotCRP,…

- - - - Q: give me one cause of training straggler.
        
        Q: Any ideas to mitigate such stragglers?
  - - - three classes of the questions
        
        How bad/good it is?
        
        Any patterns in practice?
        
        Root cause?
- - - - reduce-scatter
      - all-gather
  - - - sync vs. async computation
      - same-length vs. variable-length stages
      - single-thread vs. parallel jobs
      - explicit vs. implicit dependency
      - static vs. dynamic computation graph