Please enable JavaScript.

Coggle requires JavaScript to display documents.

LLM Systems Seminar (CS7670): Week08a, KV Cache (paper, HotCRP, lottery) -…

- - - - without KV Cache, step-by-step
      - with KV Cache, step-by-step
    - - Borrowed from UDL book
      - Borrowed from Distributed Representations of Words and Phrases and their Compositionality, representations of words
- - - - CPU cache
      - OS page cache
      - CDN (web caches)
    - - items (K/V) are the same size and known ahead-of-time
      - items are always reused
      - the reusing happens at regular time intervals (auto-regression)
  - - - cache admission and eviction
      - decide which objects to keep or evict to maximize hit rate or utility.
      - InfiniGen: Efficient Generative Inference of LLMs with Dynamic KV Cache Management
    - - multi-tier/hierachical caching
      - trade-off: size vs. fetching latency
      - research topics: coordination between levels, prefetching, replication strategies, and balancing between latency and bandwidth
      - FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
    - - problem: the consistency across multiple copies of caches
      - examples: CPU cache coherence protocol
    - - Concerns: Side-channel attacks (timing, eviction-based), cross-tenant leakage in shared caches
      - Research directions: Cache partition isolation, randomization, oblivious caching
      - PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption
    - - core: bring "correctness" (or model accuracy) into the trade-off
      - Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
      - H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
      - DiffKV : Differentiated Memory Management for Large Language Models with Parallel KV Compaction (SOSP'25)
- - - - within GPU
      - outside GPU
    - - think of this as a database
      - multi-turn conversations
- - - - Pro side: Yes, systems serve users, not equations.
      - Con side: No, it violates computational integrity.