A support bot, a RAG pipeline, and a coding agent all send the model context it has already processed: the same system prompt, the same retrieved contract, the same history plus one new turn. vLLM can reuse that work while the blocks are still on one replica’s GPU. Once they’re evicted, or the request lands on another replica, the engine runs prefill again for tokens it already paid for.
LMCache is the open-source layer that keeps KV state available outside GPU memory for later reuse. We were founded by its creators and build on it; LMCache remains open source. Its docs and project site provide installation guides. This guide explains the cache path, vLLM integration, and deployment tradeoffs.
What is LMCache?
LMCache is an open-source KV cache management layer for LLM inference. It takes the KV cache a serving engine such as vLLM produces, stores it outside GPU memory in CPU memory, local disk, or remote storage, and loads it back when the same context returns. Reuse works across requests, sessions, and engine instances, which cuts repeated prefill.
The code lives in the LMCache GitHub repo, which has more than 10,000 stars and is under the Apache 2.0 license. It came out of systems research at the University of Chicago, and the LMCache paper’s authors include Tensormesh co-founders Yihua Cheng, Kuntai Du, and Junchen Jiang.
In October 2025, it became a PyTorch Ecosystem project. LMCache remains an independent open-source project, with development supported in part by Tensormesh. Its vendor-neutral design lets compatible serving engines and storage systems reuse stored KV state, within the compatibility limits covered below.
The problem LMCache solves: recomputed prefill and full GPU memory
The KV cache is the record of prefill: key and value tensors for every prompt token, expensive to build and held in scarce GPU memory. Our guide to how the KV cache works covers the basics.
vLLM already reuses some of it. Automatic prefix caching lets a new request skip computing a prefix it shares with an earlier one, but those blocks live in that instance’s GPU memory. Once a cached block is popped from the head of the free queue for a new allocation, it is evicted, and no request can reuse it.
Reusable KV state can outgrow GPU capacity, making storage outside the device necessary for reuse across requests and engines. Retention alone isn’t enough, though: changing a prompt’s beginning can break prefix matches. The LMCache paper observed that context truncation reduced prefix-cache hits in real usage, so an application’s context policy can undermine reuse before the cache layer sees the request.
The savings land in prefill. A hit skips recomputing the prompt, but decode still generates every output token, so long answers gain less than long, repeated context.
How LMCache works
LMCache attaches to the engine through a connector that taps into vLLM’s paged KV memory manager. For each prompt, the connector checks whether a token sequence has been seen before. On a hit, it fetches the previously computed chunks from the cache; on a miss, the model computes KV as usual and the connector hands the new chunks over for storage. The index behind that lookup maps token sequences to cached chunks of 256 tokens by default.
Tiered storage: GPU memory, CPU memory, local storage, remote storage
Moving KV out of GPU memory into these tiers is KV cache offloading, with capacity and transfer latency depending on the tier and backend. In the multiprocess (MP) architecture, LMCache calls CPU memory its L1 cache and the persistent storage behind it L2.
| Tier | What it holds | Name in LMCache MP mode |
|---|---|---|
| GPU memory | The active working set, in the engine’s paged KV cache | Engine-owned; the source of store, the target of retrieve |
| CPU memory (DRAM) | A hot cache of recently used chunks in pinned memory | L1 |
| Local disk | A larger local tier, for example for long documents | L2 |
| Remote storage | Shared or remote backends such as Redis/Valkey, Mooncake or S3-compatible object storage; latency and durability depend on the backend and how it’s configured | L2 |
That naming collides with processor caches: here L2 is the storage tier behind CPU memory, not on-chip cache.
Data moves in four operations: store copies new chunks from the GPU to L1, and retrieve copies them back on a hit. Behind L1, an async write pushes chunks to L2 without blocking inference, and prefetch pulls chunks from L2 back into L1 when a lookup misses L1. The L1 eviction policy is a required server flag with no default; the commands below use LRU.

Storage mode vs transport mode
Everything above is storage mode: LMCache acts as a persistent KV store for reuse across queries and sessions, and saved chunks can survive process restarts when the L2 backend is configured to keep and recover them. Transport mode serves prefill/decode disaggregation instead: LMCache moves the prompt’s KV from prefill workers to decode workers, which generate without recomputing it, over NVLink, RDMA, or TCP through transports such as NIXL.
Beyond prefix matching: non-prefix reuse with CacheBlend
Prefix caching only helps when the reused text is the prefix. RAG breaks that: a retrieved chunk lands in a different position each time, and its cached KV ignores the cross-attention with whatever now precedes it. CacheBlend, published at EuroSys 2025 by the same research group, reuses a repeated chunk’s KV at any position and selectively recomputes a small fraction of tokens to recover quality.
LMCache and vLLM: how they fit together
vLLM, the serving engine, schedules requests, runs the model, and manages GPU memory. LMCache is the cache layer it plugs into through its KV connector. The two caches cooperate: if vLLM still holds part of a matching prefix on the GPU, the connector only needs to retrieve the missing portion. In the LMCache quickstart example, 16 tokens of a 24-token hit remain on the GPU, so LMCache supplies the other 8. Keeping that boundary avoids loading state the engine already has.
Two deployment modes
LMCache supports two deployment modes with vLLM, and MP mode is the recommended one: it runs as a standalone server, vLLM attaches through LMCacheMPConnector, and one cache can be shared across engine instances. Legacy in-process mode runs LMCache inside vLLM through LMCacheConnectorV1 for single-node experiments. MP mode also separates failure domains, since a cache problem doesn’t crash the engine and an engine crash doesn’t lose the cache.
These commands follow the same quickstart as of October 3, 2026 (lmcache 0.5.5 on PyPI), minus its demo chunk size so the server keeps the default:
uv pip install lmcache vllm
# Terminal 1: start the LMCache server
lmcache server --host localhost --port 5555 \
--l1-size-gb 20 --eviction-policy LRU
# Terminal 2: start vLLM with the LMCache-shipped connector
vllm serve Qwen/Qwen3-8B --port 8000 --kv-transfer-config \
'{"kv_connector":"LMCacheMPConnector",
"kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector",
"kv_role":"kv_both",
"kv_connector_extra_config": {"lmcache.mp.host": "localhost", "lmcache.mp.port": 5555}}'The kv_connector_module_path setting needs vLLM 0.20.0 or newer. It makes vLLM load the connector LMCache ships instead of the copy vendored into vLLM, which can trail the LMCache server protocol. On older vLLM versions, leave it out to get the bundled connector. Pin both versions before you deploy, because a new vLLM release can change the KV cache APIs the connector uses, and check the quickstart for environment and ABI prerequisites first.
Other engines, as of October 2026
Support beyond vLLM is at different stages of maturity. The SGLang integration runs in MP mode only and needs SGLang, and LMCache builds that include specific upstream pull requests. SGLang keeps its GPU radix cache and calls the LMCache server for lookup, retrieval, and storage. The TensorRT-LLM integration depends on changes not yet in a stable release, so both install from source for now. NVIDIA Dynamo documents an LMCache integration for its vLLM backend, on x86 only.
What LMCache is used for
Cache reuse happens wherever the same tokens come back:
| Workload | What repeats | What LMCache does with it |
|---|---|---|
| Multi-turn chat | The whole conversation history on every turn | Keeps earlier turns’ KV so the next turn prefills only the new message |
| RAG and long-document analysis | The same documents and chunks across users and questions | Stores chunk KV once; CacheBlend reuses it even when the chunk isn’t the prefix |
| Agent loops | System prompt, tool definitions and growing context, every step | Serves the stable part from cache across long sessions |
| Prefill/decode disaggregation | KV a decode worker needs from a prefill worker | Transfers it instead of recomputing |
| Multi-instance fleets | Shared context hitting different replicas | Shares one cache across instances on a node, and across nodes with P2P |
The LMCache paper reports up to 15x higher throughput combining LMCache with vLLM on multi-round question answering and document analysis, testing version 0.3.6 on H100 GPUs against baselines that include vLLM’s GPU-only prefix caching.
Oracle’s OCI Data Science write-up ran a multi-turn conversation benchmark with 5,000-token inputs on Granite 3.3 8B, with LMCache in-process and caching to CPU memory. Median time to first token (TTFT) fell from 18.5 seconds without LMCache to 8.0 seconds with a hot cache. Oracle attributes the drop to hits on the common prefix and earlier turns. Those hits happen within a single run, so most of the gain showed up on the cold-cache run too. Neither is a planning number for your deployment.
LMCache vs other ways to cache LLM work
The usual “alternatives” to LMCache are different layers caching different things:
| Approach | What is reused | Where it lives | Scope | Works with hosted APIs? |
|---|---|---|---|---|
| Engine prefix caching (vLLM APC) | KV blocks for an identical prefix | GPU memory of one instance | One instance, until LRU eviction | No, it is part of a self-hosted engine |
| LMCache | KV cache chunks, prefix or non-prefix with CacheBlend | CPU memory, local disk, remote backends | Requests, sessions and engine instances | No, it plugs into an engine you run |
| Hosted prompt caching | A prompt prefix up to a cache breakpoint | The provider’s infrastructure | Your calls to that provider | Only for models behind that API |
| Semantic caching | Earlier answers matched by meaning | An application-side or gateway store | Similar questions; the model is skipped on a hit | Yes, it sits in front of any model call |
Anthropic’s prompt caching reuses a processed prefix inside the provider’s API; hosted-model reuse depends on that provider’s caching features. Semantic caching returns a stored answer without calling the model, so it works on text rather than reusing KV tensors. LMCache uses systems such as Mooncake and Redis for storage and NIXL for transfers.
Running LMCache in production (and where Tensormesh Platform fits)
LMCache is production software in its own right. The project ships a Kubernetes operator: declare an LMCacheEngine resource, and it reconciles the objects underneath. It also exports KV cache metrics, prefix cache hits included, through OpenTelemetry and Prometheus, and offers P2P sharing that turns per-node caches into one logical cache. Compatibility needs its own deployment checks: a version pair alone doesn’t establish support for every model, connector, device, and storage backend. Check runtime ABI, connector loading, and model or feature support separately.
Tensormesh Platform isn’t a serving engine and doesn’t replace LMCache: we built it on the project our founders created, and it adds the enterprise management layer for teams running inference at scale in production. It manages three tiers: L0 GPU memory, L1 host memory, and L2 filesystem-backed storage, so its L1 and L2 match LMCache’s names. State saved outside an inference pod can be reused after that pod restarts. Cross-node sharing, which builds on LMCache’s peer-to-peer architecture, lets a node retrieve compatible cached work from another node instead of recomputing it. Helm handles installation; a web UI and CLI cover day-to-day operations, observability, and multi-tenancy; and the platform supports prefill/decode disaggregation and CacheBlend non-prefix reuse for dense models.
Its scope is narrower than LMCache’s: Tensormesh Platform runs with vLLM on Kubernetes, on NVIDIA or supported AMD Instinct GPUs through ROCm, with filesystem-backed storage, and SGLang support is next. It runs in your environment and doesn’t attach to a model you only reach through a hosted provider’s API.
Run LMCache yourself if you want full control and have the team to own version pinning, storage backends, and upgrades. Look at Tensormesh Platform if you serve vLLM on Kubernetes and want managed tiers, cross-node sharing, and multi-tenant operations without building that layer. Measure hit rate alongside reload time and prefill latency to see whether reuse pays for the transfer cost.
To try LMCache, start with the quickstart. If you run vLLM on Kubernetes and want the production layer managed, contact our team.
Frequently asked questions
What does LMCache do?
LMCache stores the KV cache an inference engine has previously computed outside GPU memory and loads it back when the same context returns. By skipping repeated prefill for chat history, shared documents, and agent context, LMCache reduces TTFT and raises throughput.
Is LMCache open source, and where is the GitHub repo?
Yes. LMCache is Apache 2.0 licensed. You can get the code from the LMCache GitHub repository or install the package with
pip install lmcache.Does LMCache work with SGLang?
Yes, with conditions: as of October 2026, the integration runs in MP mode only and needs SGLang and LMCache versions with specific upstream changes, so check the quickstart first. Tensormesh Platform supports vLLM today; SGLang support is next.
What is the difference between LMCache and vLLM?
vLLM is the serving engine that runs the model and manages GPU memory. LMCache is the KV cache layer that plugs into it through a KV connector and keeps reusable cache in CPU memory, disk, and remote storage, beyond what vLLM holds on the GPU.
Who created LMCache?
LMCache came out of systems research at the University of Chicago, and its paper’s authors include Tensormesh co-founders Yihua Cheng, Kuntai Du, and Junchen Jiang. It remains an independent open-source project.
Does LMCache work with hosted APIs like OpenAI or Anthropic?
No. LMCache plugs into an engine you run, because it needs that engine’s KV cache. Tensormesh Platform has the same boundary and doesn’t attach to a model accessed only through a hosted provider’s API; there, the provider’s own prompt caching applies.