Learn

LLM inference & KV cache guides

Practical guides on LLM inference, KV caching, GPU efficiency and serverless deployment: what you need to cut the cost of running AI in production.

● Start here · three guides, in order

By Matt Tanner, Developer Relations

Step 1 of 3

LLM Inference: How It Works and How to Run It in Production

A practical guide to LLM inference: how it runs in production, the tuning knobs that actually matter, and the pitfalls that inflate cost and latency.

Guide · Fundamentals

Step 2 of 3

KV Cache: What It Is and Why It Costs You Money

KV cache is the memory system behind every fast LLM response, and the reason your inference bill is what it is. Here's how it actually works.

Guide · Fundamentals

Step 3 of 3

Prefill vs. Decode: The Two Phases of LLM Inference

Learn the difference between prefill and decode in LLM inference: why one is compute-bound, one is memory-bound, and how the KV cache links them.

Guide · Fundamentals

Library

Every guide, by topic

KV cache

View all →

Semantic Caching: How It Works, and Where It Stops

How to Calculate KV Cache Size: A Step-by-Step Sizing Method

KV Cache Quantization: FP8, FP4, and What They Cost You

KV Cache and Attention: The Mechanics Explained

KV Cache Memory: Which Lever to Pull, and When

LLM-Based Chunking: What It Is & How It Works

Serving & engines

View all →

vLLM’s OpenAI-Compatible Server: What Transfers and What Doesn’t

Disaggregated Serving in LLM Inference, Explained

How to Serve LLMs Efficiently Across Multiple Workloads

What Is vLLM? A Plain-English Guide to the Open-Source Engine

vLLM vs SGLang (2026): Which Should You Choose?

LLM Serving: A Complete Guide to Production Inference

Cost & deployment

View all →

Self-Hosted LLM: What Running Your Own Stack Actually Costs

Top Tools to Reduce ML Inference Costs in 2026

Serverless Inference: When It Works, When to Leave It

Anthropic Prompt Caching Pricing: The Full 2026 Breakdown

What Actually Drives LLM Inference Cost (And How to Cut It)

Worked example

Size the cache before you buy the GPUs

Every token you keep costs memory in every layer. Multiply it out and you know how much context fits on a card, and how much needs a cheaper tier.

KV bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_value

layers

kv_heads

head_dim

128

FP16 bytes

Example: 8B-class model with grouped-query attention

128 KiB / token

At 32,000 tokens of context that is 4 GiB for a single sequence, before batching.