Please enable JavaScript.

Coggle requires JavaScript to display documents.

LLM Systems Seminar (CS7670): Week03b, LLM serving (lottery) - Coggle…

- - - - often, cheaply and quickly are in tension
  - - - many "new" systems
  - - - RPS (Request per second)
      - TPS (Tokens per second)
      - request latency
      - TTFT (Time-to-First-Token)
      - TPOT (Time-per-Output-Token) or ITL (Inter-token Latency) or TBT (Time-between-Tokens)
      - SLO (Service level objective)
- - - - LLM <-> DL
  - - - gif
- - - - Llumnix: Dynamic Scheduling for Large Language Model Serving, OSDI'24
        
        Llumnix dynamically reschedules incomplete inference requests across model instances using a context‑switch‑like mechanism.
      - ServerlessLLM: Low-Latency Serverless Inference for Large Language Models, OSDI'24
        
        recompute (rather than state migration)
      - LLM<->DL
    - - SpotServe: Serving Generative Large Language Models on Preemptible Instances, ASPLOS'24
        
        supports LLM inference using preemptible instances for improving cost efficiency
      - LLM<->DL
      - FastServe, Fast Distributed Inference Serving for Large Language Models
        
        optimizes request completion times using a preemptive time-slicing approach
    - - FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, ICML'23
        
        model weight offloading
      - InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management, OSDI'24
        
        Offloading‑based inference systems suffer when KV caches overflow GPU memory.
      - LLM<->DL
    - - VTC, Fairness in Serving Large Language Models, OSDI'24, Serving fairness
      - LLM<->DL
  - - - Tabi uses small models for most queries and calls an LLM only for hard cases; calibrated confidence guides the fallback.
      - LLM <-> DL
  - - - Borrowed from Unlocking Efficiency in Large Language Model Inference:A Comprehensive Survey of Speculative Decoding