Reasoning models made your traffic decode-heavy: 30k thinking tokens per request. What changes in your serving stack?

When every request thinks for 30,000 tokens, serving flips from compute-bound prefill to memory-bound decode, and the KV cache becomes the resource you actually schedule. The levers that ruled chat traffic stop being the ones that matter.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.