← 🧠 Foundations of LLMs & GenAINEXT IN FOUNDATIONS OF LLMS & GENAILoRA and Parameter-Efficient Fine-Tuning→
Core
The KV Cache
During autoregressive decoding, a model would recompute attention over the entire history at every step; the KV cache stores each token's key and value vectors so each new token only attends, never recomputes. The win is compute; the cost moves to memory: the cache grows with sequence length times batch size and usually becomes the binding constraint in serving. Applied-AI interviews probe it because it explains why long contexts are expensive to serve, why throughput (not model speed) is often the limit, and why MQA/GQA and PagedAttention exist.
a free account unlocks the core curriculum tier · no card
COURSES COVERING THIS TOPIC
No lesson covers this one directly yet. These teach the surrounding topic from the beginning.
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
ML Infrastructure & GPUsServe a 70B-parameter model with high throughput. Do the memory math and name the optimizations.→ML Infrastructure & GPUsExplain the KV cache: prefill vs decode, why it grows, and how MQA/GQA and PagedAttention help.→ML Infrastructure & GPUsExplain speculative decoding and the other main levers for cutting LLM generation latency.→LLM & GenAI FundamentalsWhat are Multi-Query (MQA) and Grouped-Query Attention (GQA), and why do they exist?→ML Infrastructure & GPUsYour inference p50 is fine but p99 latency spikes under load. How do you fix tail latency?→ML Infrastructure & GPUsHow do you load-test and capacity-plan an LLM inference service before launch?→
COMPANIES THAT ASSUME THIS