By Chitral Patil
As generative AI moves into production, enterprises face an important infrastructure decision: consume models through commercial APIs, deploy open-weight models on dedicated GPUs, or combine both approaches?
The comparison is often reduced to simple arithmetic. Divide the hourly cost of a GPU by a model’s maximum token throughput and compare the result with an API provider’s per-token price. By this method, self-hosting frequently appears dramatically cheaper.
But a provisioned GPU does not automatically operate at benchmark throughput.
Why enterprises self-host
Cost is not the only reason to choose dedicated infrastructure. Some workloads require tighter control over where data is stored and processed, who can access it, how models are configured, and how systems are audited. Privacy, data sovereignty, customisation and operational control can justify self-hosting even when its raw per-token cost is not the lowest.
These benefits are real, but they should not obscure the economics. The relevant question is what the enterprise is paying for those benefits under its actual workload.
Utilisation is the missing variable
In my research paper, Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation, I studied how offered request rate and the resulting in-flight concurrency affect the effective cost of self-hosted inference.
Across experiments on identical H100 hardware, effective cost ranged from $0.21 to $15.25 per million output tokens. At low-to-moderate enterprise loads, underutilisation increased costs by 2.5x to 24x, reaching 36.3x near idle.
A useful analogy is a scheduled bus. The operator pays for the vehicle, driver and journey whether it carries one passenger or forty. When the bus is full, the fixed cost is distributed across many passengers.
When it departs nearly empty, the effective cost per passenger is much higher.
Latency objectives create a similar constraint in LLM serving. An operator could briefly hold incoming requests to assemble more work, just as a bus could wait for additional passengers. But a strict time-to-first-token or end-to-end latency objective limits that wait. The server may need to begin processing with only a few active requests, leaving some of the GPU’s economic capacity unused.
Modern serving systems use continuous batching to add and remove requests dynamically. This improves efficiency, but it cannot manufacture concurrency when traffic is absent.
One experiment shows the result clearly. A Mixtral 8x7B FP16 deployment cost $15.25 per million output tokens at one request per second. At 25 requests per second, the same deployment cost $0.87. The hardware and model had not changed; the fixed expense was simply distributed across more completed tokens.
This workload-level qualification is also reflected in recent infrastructure research. In a July 2026 analysis of AI industry economics through 2030, Satoshi Matsuoka, director of the RIKEN Center for Computational Science, cited the measured range and similarly noted that self-hosting is not automatically cheaper unless utilisation is sufficiently high.
Self-hosting has a cost curve
Self-hosting does not have one universal cost per token. Its cost curve is shaped by hardware, model architecture, quantisation, request rate, request lengths, traffic burstiness and latency objectives.
Dedicated infrastructure can deliver compelling economics at predictable utilisation. But an internal AI assistant used continuously by thousands of employees has a different cost profile from a specialised application receiving sporadic requests, even when both use the same model and GPU.
Privacy and sovereignty requirements may still make dedicated infrastructure the right choice for the specialised application. In that situation, the enterprise should recognise that it is paying for greater control, not assume that self-hosting is automatically the cheapest option.
The case for hybrid infrastructure
For many enterprises, the most economical answer may be a hybrid architecture.
Stable base demand, sensitive workloads or applications requiring greater control can be routed to dedicated deployments. Commercial APIs or serverless endpoints can handle experiments, unpredictable bursts, overflow traffic and workloads that cannot keep a dedicated GPU busy.
I explored this operational gap in the GEICO Tech article, When “Cheaper” LLM Hosting Gets Expensive. Per-token APIs make consumption visible because every request is metered. Self-hosted infrastructure can hide the same unit economics inside a recurring hourly GPU bill.
To make that hidden cost observable, I released vllm-cost-meter, an open-source tool that connects to a live vLLM server and calculates effective dollars per million tokens from the workload it is actually processing.
A calculator predicts cost from assumptions. A meter reports cost from observed behaviour.
Questions technology leaders should ask
Before approving a dedicated LLM deployment, technology leaders should examine expected request rates, the concurrency generated by that traffic, and the spare capacity required to satisfy latency objectives.
They should also separate two decisions. First, whether privacy, sovereignty, customisation or operational control requires dedicated infrastructure. Second, whether the resulting workload can utilise that infrastructure economically.
Maximum-throughput benchmarks describe what a system can achieve at its limit, not necessarily what the organisation will pay under its actual workload.
The better question is not simply, “How much does this GPU cost?”
It is, “At our traffic level and service objective, how much useful work will this GPU complete, and what additional control are we receiving for the cost?”
Until those questions are answered using production measurements, any claim that self-hosting is cheaper remains incomplete.
-Chitral Patil is an Engineer II at GEICO, where he works on large language model inference and infrastructure. He is the solo author of Beyond Per-Token Pricing and the creator of the open-source vllm-cost-meter project.