Once you run an LLM on your own GPUs, capacity planning becomes an operating responsibility. The weights may fit, but the context length and concurrency your application needs may not. A serving engine’s startup measurements help expose that limit; for example, vLLM can report:
INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens
INFO 07-23 13:56:04 [kv_cache_utils.py:779] Maximum concurrency for 40,960 tokens per request: 15.70x
Those figures connect available memory to the requests the deployment can support. This guide follows the rest of the operating cost: hardware requirements, model licensing, multi-node networking, monitoring, and repeated computation. It assumes you already control the weights and hardware and need to understand what running the stack demands.
What self-hosting an LLM actually commits you to
Self-hosting an LLM means running the model weights on hardware you control, behind a serving engine you operate, with an inference endpoint your applications call directly. The commitment isn’t buying the GPU. It’s a standing obligation: a Linux fleet, a pinned engine version, a license you have to comply with, and a set of metrics somebody has to watch.
If you’re comparing self-hosting with a managed API, the break-even depends on your workload; see serverless inference and when to leave it. Here, the focus is the ongoing cost of a stack you already operate.
Only half of that obligation is denominated in dollars per GPU-hour. The other half is engineering surface: every prerequisite, every flag whose default is wrong for you, every failure mode that only appears at four nodes. That half is knowable in advance.
The install has a hard floor, and it isn’t a model size
Start with what the engine requires, the one part of self-hosting with a published, non-negotiable answer.
The configuration examples use v0.28.0. Pin a tested release for production and check its GPU installation requirements:
NVIDIA GPU with compute capability 7.5 or higher, such as T4, RTX20xx, A100, L4, H100 or B200.
That is a floor for running vLLM at all, not a floor for running any particular model, and the difference is where most hardware advice about self-hosted LLMs goes wrong. A T4 clears the compute-capability bar. It will not hold a 70B model.
The documented software requirements are Linux and Python 3.10–3.13, with precompiled C++ and CUDA 12.9 binaries. Blackwell GPUs require at least CUDA 12.8, including compatible PyTorch wheels. Building from source requires GCC/G++ 11.3 or newer.
CUDA and ROCm deployments require Linux. vLLM has no native Windows support; its documentation points to WSL or community forks. Apple Silicon uses the separate vLLM-Metal plugin, which requires macOS 15 or later and native arm64 Python 3.12. CPU deployment has its own installation path and limitations.
Containerized, the launch is a single documented command:
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6BConfigure container shared memory using –ipc=host or –shm-size, especially for tensor parallel inference. vLLM uses it for inter-process communication, and insufficient shared memory can cause failures that appear to be model-related.
Outside a container, the quickstart is one line: vllm serve Qwen/Qwen2.5-1.5B-Instruct. That starts a small model on one card once vLLM and its dependencies are installed.
Why VRAM requirements depend on the deployment
VRAM requirements depend on more than parameter count. Quantization, model architecture, context length, and concurrency all affect capacity, so measure the deployment you plan to run rather than relying on a generic sizing table.
A useful VRAM sizing table must state the quantization scheme, context length, batch size and KV cache budget. Changing any of these changes the memory requirement.
vLLM’s parallelism and scaling guide provides a procedure for measuring capacity:
Use one GPU if the model fits. If it needs multiple GPUs within one node, use tensor parallelism. If it exceeds one node, combine tensor and pipeline parallelism.
Add GPUs or nodes until the available memory can accommodate the model.
The measurement is the pair of log lines at the top of this article. vLLM explains what they mean:
GPU KV cache size reports how many tokens the GPU cache can hold at once. Maximum concurrency estimates the number of simultaneous requests at the specified token length. Use these measurements to check capacity and add GPUs or nodes when necessary.
That beats any table, because it’s measured on your model, your quantization, your context length, and your card. Compare GPU KV cache size with the combined prompt and output lengths you expect, then validate Maximum concurrency under your target request rate and latency. Both print for free at every start.
Two model-specific examples illustrate the role of quantization. OpenAI’s gpt-oss-120b has 117B total parameters and 5.1B active parameters, while gpt-oss-20b has 21B total and 3.6B active. Their model card reports that MXFP4 quantization of the MoE weights allows the larger model to fit on a single 80 GB GPU and the smaller one within 16 GB of memory.
A 117B-parameter model on one 80GB card is a statement about a specific quantization of a mixture-of-experts architecture, not a general rule about 100B-class models. Keep the MXFP4 condition attached to that memory figure.
The underlying KV cache sizing math sits in our LLM serving guide. Use it for direction, and your own log lines for the number.
One model per server, and what that costs you
The vLLM quickstart describes the basic server deployment:
vLLM exposes an OpenAI-compatible server with model-listing, chat-completion, and completion endpoints. Each server currently hosts one base model.
The one-model-per-server limit affects capacity planning. A chat model, an embedding model, and a small classifier mean three servers, three sets of weights in GPU memory, three health checks, and three upgrade paths. Consolidation is not something self-hosting gives you for free, and the capacity covering each peak idles between peaks.
Protocol compatibility does not guarantee identical parameter behavior. vLLM does not support suffix on /v1/completions and ignores user on /v1/chat/completions, so test any application behavior that depends on those fields. We cover the endpoint-level differences in what transfers to vLLM’s OpenAI-compatible server.
The license is an operating constraint, not a footnote
Model selection gets discussed as a benchmark question and settled as a legal one. The two most commonly deployed open-weight families are not licensed the same way, and treating them as interchangeably permissive is the commonest error in writing about self-hosted LLMs.
Qwen3-30B-A3B is Apache 2.0. The model card’s own metadata says so, and it links to the license file. No display obligation, no naming obligation, no user-count trigger, no incorporated use policy.
Llama 3.1 is open-weight, not open-source. Its license file is titled “LLAMA 3.1 COMMUNITY LICENSE AGREEMENT” and it is not OSI-approved. The grant is real and commercially usable: a “non-exclusive, worldwide, non-transferable and royalty-free limited license” to “use, reproduce, distribute, copy, create derivative works of, and make modifications to the Llama Materials.” Four obligations Apache 2.0 does not impose come attached, and each lands on a different team.
The first is distribution. Ship the materials or a product containing them and you “shall (A) provide a copy of this Agreement with any such Llama Materials; and (B) prominently display ‘Built with Llama’ on a related website, user interface, blogpost, about page, or product documentation.” Fine-tune and distribute the result and “you shall also include ‘Llama’ at the beginning of any such AI model name.” A marketing-site change and a naming convention, decided by a licence.
The second is an acceptable use policy incorporated by reference: your use must “adhere to the Acceptable Use Policy for the Llama Materials,” which is “hereby incorporated by reference into this Agreement.” Apache 2.0 has no use restrictions, so this is compliance work that doesn’t exist on the Qwen side.
The third is a scale gate at “700 million monthly active users in the preceding calendar month” as of the version release date, above which “you must request a license from Meta, which Meta may grant to you in its sole discretion.” Most companies never touch it. The ones that might should know first.
The fourth is a broader termination trigger. Apache 2.0 section 3 terminates only the patent license, and only for a patent claim against the work itself. The Llama clause terminates “any licenses granted to you under this Agreement” if you institute proceedings “against Meta or any entity” alleging the materials or their outputs infringe IP rights you own or can license. That belongs with your patent strategy, not in a model bake-off.
Both licenses let you own your fine-tuned derivatives and both permit commercial deployment. They are not the same instrument, and “open-weight” is the accurate word for the Llama family.
Configured context length and usable capacity
Qwen3-30B-A3B supports 32,768 tokens natively. Its model card reports validation up to 131,072 tokens using YaRN, so the larger context window requires an extended-context configuration.
Enable the larger window with a RoPE scaling configuration at launch. The model card warns that static YaRN keeps its scaling factor fixed across input lengths and can affect performance on shorter texts, so enable it only when long contexts are required.
So the 131K number is real, validated, and not free. A capacity plan built on it assumes a serving configuration you have to enable, on a model whose own authors say enabling it can degrade short-prompt quality. The card adds that the default max_position_embeddings in config.json is 40,960, allocated as 32,768 output tokens and 8,192 tokens for typical prompts. Three numbers, three meanings, one model. Quote the one you actually configured.
Operating a multi-node deployment
Everything above fits on one machine. The cost changes shape when the model doesn’t.
vLLM defaults to Ray for multi-node inference and Python multiprocessing within a single node. Ray is an optional dependency that you must install explicitly. Every node needs the same execution environment, including Python packages and model paths; weights must be available at the same path on each node or through a shared filesystem.
The network guidance is:
“For security, set VLLM_HOST_IP to an address on a private network segment. Traffic sent over this network is unencrypted, and the endpoints exchange data in a format that can be exploited to execute arbitrary code if an adversary gains network access. Ensure that untrusted parties cannot reach the network.”
The inter-node channel is unencrypted, and the guidance is network isolation rather than transport security, which in a regulated environment means a compliance conversation, an architecture review, and probably a dedicated VLAN.
Efficient cross-node tensor parallelism needs a fast interconnect, preferably InfiniBand. In vLLM’s NCCL logs, [send] via NET/IB/GDRDMA indicates InfiniBand with GPUDirect RDMA, while [send] via NET/Socket indicates raw TCP, which is inefficient for cross-node tensor parallelism.
Grep for that on your first bring-up. It explains a whole class of disappointing benchmarks.
One topology detail saves money. If the GPU count doesn’t evenly divide the model, or the node has no NVLink, vLLM recommends pipeline parallelism over tensor parallelism to improve throughput and reduce communication overhead. Cheaper cards without NVLink are a legitimate strategy, just not on the parallelism setting most tutorials show.
Measure the waste before you buy hardware to cover it
The highest recurring cost here isn’t the one on the invoice. It’s the GPU-hours spent recomputing context that hasn’t changed since the last request: the same system prompt, the same tool schemas, the same retrieved documents, the same history replayed on every turn of an agent loop. A long fixed prefix gets paid for repeatedly, because the KV cache that work produces is discarded between requests unless something keeps it. This is how the KV cache works under the hood, and how it affects the energy picture in our analysis of inference costs and the energy constraint.
You can measure prefix reuse through the /metrics endpoint on vLLM’s OpenAI-compatible server. Two counters track it directly:
vllm:prefix_cache_hits counts cached tokens reused; vllm:prefix_cache_queries counts tokens queried for reuse.
Divide the first by the second to get your prefix cache hit rate in cached tokens on your own traffic. That beats any external benchmark, because prefix reuse is a property of your prompts rather than of the engine.
Three more metrics help diagnose capacity. vllm:kv_cache_usage_perc reports cache utilization, with 1 representing 100%. vllm:num_preemptions counts requests preempted by the engine. vllm:request_prefill_kv_computed_tokens records new KV tokens computed during prefill, excluding cached tokens.
Metrics also require version maintenance. A metric deprecated in X.Y is hidden in X.Y+1, where –show-hidden-metrics-for-version=X.Y can restore it, and removed in X.Y+2. Update dashboards and alerts before that removal.
Cutting the bill without ripping out your stack
Caching fixes none of the above. Hardware sizing, electricity and the ops surface stay where they are. It addresses one line item, the duplicated prefill, and that’s the problem we built Tensormesh for, as the team behind the open-source LMCache project. Tensormesh Platform supports vLLM, and our compatibility matrix provides model and configuration details.
You can add context reuse to an existing vLLM deployment. Our existing-deployment guide describes the required pod changes: an image with the lmcache package installed, the engine connection ConfigMap mounted and passed to –kv-transfer-config, hostIPC: true for MP-mode shared memory, PYTHONHASHSEED=0 for deterministic token hashing, and scheduling onto a node that also runs an LMCache engine pod.
If a prerequisite is missing, the deployment may start while LMCache fails or silently uses the wrong connector path. Budget a deployment change, not a migration. The Platform runs on Kubernetes with NVIDIA and AMD GPUs and installs with Helm.
Our separate Serverless Inference service provides cached tokens at $0.00 per million on most models. The available savings depend on how much context your workload reuses; check the rate card for the model you plan to run.
Either way, the return scales with how repetitive your prompts are, which is why agent and RAG workloads see the most and one-shot classification the least. We make the full argument in cutting inference cost through context reuse and the hidden cost of open-weight LLM serving.
What none of this reduces to a number
Some costs refuse to become a line item, which is how TCO spreadsheets end up wrong in the same direction every time.
Staffing concentration is the first. The prerequisites above describe a stack with CUDA versions, compiler constraints, NCCL transport behavior, RDMA capabilities and a Ray cluster in it, and in most teams that knowledge lives in one or two people. That’s not a reason to avoid self-hosting. It’s a reason to write the runbook while the person who has it is still there.
Version drift is the second. vLLM published seven releases between June 29 and August 26, 2026; metrics deprecate on a two-release clock, and model cards set minimums of their own: Qwen3-30B-A3B asks for vllm>=0.8.5 or sglang>=0.4.6.post1 just to serve it. Staying current is work, and not staying current is a different kind of work that arrives all at once.
The third is provisioning for a peak that rarely comes. One model per server, plus headroom for the busiest hour of the week, produces a fleet with low average utilization. That underused headroom is a real line item, and it is the one most TCO models omit. Whether it is cheaper to carry it or to burst elsewhere is the question we work through in the serverless inference guide.
Assessing the total cost
You already control the weights. That was the hard part.
Plan for hardware requirements, workload-specific memory sizing, one model per server, license obligations that differ materially between families, an unencrypted internode path, and a metrics endpoint that tells you how much of your prefill work is duplicated. The first five are costs to plan for. The last says which to attack first.
So read the two log lines your server prints, then pull vllm:prefix_cache_hits and vllm:prefix_cache_queries from /metrics for a week of production traffic and divide. If a large fraction of your prompt tokens is the same across requests, the cheapest capacity you can add isn’t another GPU.
Contact our team to discuss caching for your self-hosted LLM deployment.
Frequently asked questions
What is a self-hosted LLM?
A self-hosted LLM is an open-weight model you run on infrastructure you control, served through an engine like vLLM behind your own endpoint. You hold the weights, choose the hardware, set the serving configuration, and take on the operations: upgrades, monitoring, capacity planning, and license compliance.
Can you self-host an LLM?
Yes, and the engine layer is no longer the hard part. vLLM serves a model with one command, vllm serve Qwen/Qwen2.5-1.5B-Instruct, on a Linux host with Python 3.10 to 3.13 and a GPU of compute capability 7.5 or higher. Production adds the routing, autoscaling, availability, and observability that the engine doesn’t provide.
How much does it cost to host a self-hosted LLM?
There is no universal figure. It depends on the model, quantization, context length, concurrency target, utilization, and redundancy level you choose, and on your own hardware or cloud pricing. The inputs are yours: the GPU KV cache size and Maximum concurrency figures vLLM prints at startup, your prefix cache hit rate, and your sustained token volume. The comparison against per-token pricing is in our serverless inference guide.
How much VRAM do I need to self-host an LLM?
VRAM requirements depend on model architecture, precision, context length, and concurrency. vLLM’s guidance is to increase GPUs and nodes until the model fits, then read the GPU KV cache size and Maximum concurrency lines in the startup log and add capacity if they’re below your throughput requirements. The data points that do exist are model-specific: OpenAI’s card says gpt-oss-120b runs on a single 80GB GPU and gpt-oss-20b within 16GB, both with MXFP4 quantization of the MoE weights.
What is the best self-hosted LLM model?
Constraints decide this more often than benchmarks. Licensing is the first filter: Qwen3-30B-A3B is Apache 2.0, while Llama 3.1 ships under a Community License with a “Built with Llama” display requirement, a model-naming requirement, an incorporated acceptable use policy, and a 700 million MAU gate. Context length is the second, and native windows differ from extended ones. GPU fit is the third, and you measure that rather than look it up.