A cloud GPU pricing comparison is only half an answer until the hourly rate becomes a cost per million tokens. The conversion is GPU-hour rate × 1,000,000 ÷ (tokens per second × 3,600). Add utilization, uptime, storage, software, and operations before calling the result a production cost.
That distinction matters because a published GPU-hour price is only an input. The same H100 rents for $1.99 per hour on one provider's community tier and $4.29 per hour on another's dedicated cloud, and either can be the cheaper choice once throughput and idle time are counted. This guide gives you the current provider rates, then the method to recalculate cost per million tokens against your own measured workload. (Source: Runpod pricing, Lambda GPU cloud pricing)
The practical answer: measure tokens per second under the latency and concurrency your users need, then divide the fully allocated hourly cost by effective tokens per hour. Peak benchmark speed is not a business metric until the deployment can sustain it.
GPU cost per million tokens formula
Start with the simplest steady-state equation:
Cost per 1M tokens = hourly GPU cost × 1,000,000 ÷ effective tokens per hour
Effective tokens per hour are tokens per second × 3,600 × utilization. If you include fixed costs, replace the hourly GPU rate with an allocated rate that also covers storage, software, monitoring, support, and depreciation. NVIDIA's TCO guidance treats throughput, latency, demand, uptime, and infrastructure as connected inputs. (Source: NVIDIA inference TCO)
| Variable | Meaning | Example |
|---|---|---|
| R | GPU and allocated infrastructure cost per hour | $2.00 |
| T | Measured combined tokens per second | 120 |
| U | Sustained utilization | 50% |
| C | Cost per million tokens | $9.26 |
The example is arithmetic, not a provider quote. At $2.00, 120 tokens/s, and 50% utilization, the result is $2 × 1,000,000 ÷ (120 × 3,600 × 0.50) = $9.26. If the workload produces 80% utilization with the same speed and rate, the result falls to $5.79.
NVIDIA's inference TCO method also separates input and output token costs because output generation often has different performance and pricing behavior. Record the input/output mix instead of reporting one blended speed without context. (Source: NVIDIA inference TCO)
Cloud GPU pricing comparison: current on-demand rates
The rates below are the list prices on each provider's own pricing page, read on 2026-08-29. The right-hand column applies the formula above at a single held-constant throughput of 120 tokens per second, which is 432,000 tokens per hour at full utilization. Holding throughput constant isolates the price variable; it is not a claim that these cards perform alike. An H100 and an RTX A6000 do not produce the same tokens per second, so the real ranking changes once you substitute your own measured rate. (Source: Runpod pricing, Lambda GPU cloud pricing)
| GPU | Provider tier | $/hour | $/MTok at 120 tok/s, 100% utilization |
|---|---|---|---|
| RTX A6000 48GB | Runpod community | $0.33 | $0.76 |
| L40S 48GB | Runpod community | $0.79 | $1.83 |
| A6000 48GB | Lambda on-demand | $1.09 | $2.52 |
| A100 PCIe 80GB | Runpod community | $1.19 | $2.75 |
| A100 SXM 40GB | Lambda on-demand | $1.99 | $4.61 |
| H100 PCIe 80GB | Runpod community | $1.99 | $4.61 |
| H100 SXM 80GB | Runpod community | $2.69 | $6.23 |
| H100 PCIe 80GB | Runpod secure | $2.89 | $6.69 |
| H100 PCIe 80GB | Lambda on-demand | $3.29 | $7.62 |
| H100 SXM 80GB | Lambda on-demand | $4.29 | $9.93 |
Two things in that table decide more than the GPU choice. The same H100 PCIe card spans $1.99 to $3.29 per hour across these tiers, a 65% spread on identical silicon, because a community marketplace, a dedicated secure tier, and a managed AI cloud sell different reliability rather than different hardware. And the cheapest row is only the cheapest if the card sustains the throughput your latency target needs: an RTX A6000 at $0.76 per million tokens loses to an H100 at $4.61 the moment the smaller card cannot hold your concurrency. Price the tier and the throughput together, never separately. (Source: Runpod pricing, Lambda GPU cloud pricing)
Cloud GPU cost per hour is only the starting point
Cloud GPU cost per hour varies with region, GPU type, machine shape, operating system, commitment, and billing model. AWS and Google Cloud publish provider-specific tables, but those tables do not tell you how many tokens your model will generate. Copy the rate that matches your region and configuration, then benchmark the deployment.
The hourly line can also exclude costs that appear elsewhere on the bill. Add attached storage, snapshots, network egress, load balancers, orchestration, logging, reserved capacity commitments, and idle replicas. For a self-hosted service, include hardware depreciation, power, cooling, rack space, and the operator time needed to keep it available.
| Cost input | Include it when | Common mistake |
|---|---|---|
| GPU instance | Always | Comparing different GPU generations as if they were equivalent |
| CPU, RAM, and host | The model or runtime needs them | Counting only accelerator price |
| Storage and snapshots | Models or logs persist | Ignoring idle and backup storage |
| Network and egress | Responses leave the region or VPC | Treating transfer as free |
| Software and operations | The service runs beyond a test | Leaving monitoring and on-call out |
AWS's on-demand page and Google Cloud's GPU pricing pages are rate sources, not throughput benchmarks. Record the region, machine type, GPU count, and billing unit beside every copied price. (Source: AWS EC2 pricing, Google Cloud GPU pricing)
Cloud GPU cost comparison by utilization
Utilization is the percentage of the paid deployment's available generation capacity that produces useful tokens. It is not the CPU percentage shown in a dashboard. A queue can keep a GPU busy while a latency target is failing, so pair utilization with TTFT, intertoken latency, and request success. NVIDIA's benchmark guidance uses these latency and throughput measures together rather than treating one peak speed as the answer. (Source: NVIDIA inference TCO)
The table below holds the GPU rate at $2.00/hour and measured output at 120 tokens/s. It shows why a cloud GPU pricing comparison without utilization can mislead a buyer.
| Utilization | Effective tokens per hour | Cost per million tokens |
|---|---|---|
| 10% | 43,200 | $46.30 |
| 25% | 108,000 | $18.52 |
| 50% | 216,000 | $9.26 |
| 80% | 345,600 | $5.79 |
Add 15% overhead for storage, software, and operations and the 50% example becomes $10.65 per million tokens. That surcharge is an explicit planning assumption, not a universal cloud fee. The useful output is a range with the assumption visible.
Third-party comparison pages make the hourly-to-token conversion easier to see, but each one bakes in its own model, rate, and throughput inputs. The concurrency-aware methodology paper makes the underlying objection: a single per-token figure hides the workload shape and service constraints that produced it. Use a published comparison to find the terms, then recalculate from a rate and benchmark that match your deployment. (Source: Concurrency-aware cost paper)
Why concurrency changes GPU cost per token
Concurrency is the number of active requests being served at the same time. Low concurrency can give each request a fast response while leaving the GPU underused. Higher concurrency can improve batching and tokens per second, but it can also increase time to first token and queueing.
NVIDIA recommends measuring TTFT, intertoken latency, tokens per second, requests per second, and concurrency. Its example treats a latency limit as a constraint before selecting the highest-throughput operating point. The cheapest point on a throughput chart is irrelevant if it misses your response-time target. (Source: NVIDIA inference TCO)
The arXiv concurrency-aware methodology makes the same correction to simple per-token pricing. Infrastructure cost changes with workload shape, service constraints, and the capacity needed to handle demand. Report the concurrency used for every GPU cost per million tokens number. (Source: Concurrency-aware cost paper)
For a useful benchmark, hold the model, quantization, prompt lengths, output cap, runtime, and sampling settings constant. Run concurrency levels such as 1, 2, 4, 8, and 16. Keep the best point that meets your TTFT and error-rate limits, then calculate unit cost from that point.
Cloud GPU pricing comparison versus an API
Compare a GPU with an API at the same token mix and quality level. If the API charges $A per million combined tokens and the GPU's fully allocated hourly rate is R, the utilization needed to match the API is shown by the following equation. Provider price tables and GPU rates should be refreshed together because either side can change the break-even point. (Source: AWS EC2 pricing)
Break-even utilization = R × 1,000,000 ÷ (tokens per second × 3,600 × A)
Provider prices to test the break-even against
The table below holds the deployment at $2.00/hour and 120 tokens/s, which is 432,000 tokens per hour at full utilization, and asks what utilization each published API price demands. Prices are the list rates on the providers' own pages, read on 2026-08-27; Google's page carries a 2026-08-26 update stamp. Blended rates assume a 50/50 input-output split, which is a modeling assumption, not a provider term. (Source: Anthropic pricing, Google Gemini pricing)
| Provider and model | Input $/MTok | Output $/MTok | Blended 50/50 $/MTok | Break-even utilization |
|---|---|---|---|---|
| Claude Haiku 4.5 | $1.00 | $5.00 | $3.00 | not reachable (154%) |
| Gemini 2.5 Flash | $0.30 | $2.50 | $1.40 | not reachable (331%) |
| Gemini 3.5 Flash | $1.50 | $9.00 | $5.25 | 88.2% |
| Gemini 2.5 Pro | $1.25 | $10.00 | $5.62 | 82.3% |
| Claude Sonnet 5 | $2.00 | $10.00 | $6.00 | 77.2% |
| Claude Opus 5 | $5.00 | $25.00 | $15.00 | 30.9% |
Two rows exceed 100%, which means this deployment cannot reach those prices at any utilization: a small self-hosted instance does not beat a cheap hosted model on price alone. The frontier rows are the beatable ones. Against Claude Opus 5 the same hardware breaks even near 31% utilization, and against Claude Sonnet 5 it needs 77%. That result excludes migration cost and failover. Add those costs before choosing self-hosting.
One caveat changes the comparison and appears in no price table. Anthropic states that Claude 4.7 and later models use a newer tokenizer producing roughly 30% more tokens for the same text, while Sonnet 4.6 and earlier use the previous one, so a per-token price is not comparable across tokenizers: the same document costs more on the newer models at an identical rate. Prices also move on schedule; Google lists Gemini 3.7 and 3.6 Flash at $0.75 input and $3.75 output through 2026-12-31, then $1.50 and $7.50. Compare on cost per unit of finished work, and rerun the equation against both pages before quoting a break-even number. (Source: Anthropic pricing, Google Gemini pricing)
| Deployment choice | Usually fits | Decision test |
|---|---|---|
| API | Variable or uncertain demand | Compare current API price with no idle commitment |
| On-demand GPU | Stable bursts and a known model | Test utilization and scale-up delay |
| Reserved or committed GPU | High, predictable demand | Include commitment risk and capacity floor |
| Local or owned GPU | Data locality and steady use | Add depreciation, power, support, and failure recovery |
The break-even number is a threshold, not a verdict. An API may win at low utilization because someone else carries capacity and operations. A GPU may win at steady high utilization when the team can operate it safely. Before you accept the API price on the right side of that equation, check whether prompt caching cost reduction lowers it, because a cached prefix moves the break-even point against self-hosting.
GPU inference cost calculator inputs
An honest GPU inference cost calculator needs more than GPU name and hourly price. Capture these inputs in the same run that produces your performance number. This keeps the unit-cost result tied to a reproducible serving configuration rather than a generic hardware label. (Source: Concurrency-aware cost paper)
- Model name, version, quantization, context length, and input/output token mix.
- Runtime and serving configuration, including batching and memory settings.
- GPU type, count, region, hourly rate, and any commitment term.
- Concurrency, TTFT, intertoken latency, tokens per second, requests per second, and error rate.
- Sustained utilization, uptime target, storage, network, monitoring, and operator allocation.
NVIDIA's GenAI-Perf guidance is useful because it turns throughput and latency into infrastructure sizing inputs. The paper's contribution is equally practical: an infrastructure estimate should reflect the service conditions that make the deployment acceptable, not a single peak number. (Source: NVIDIA inference TCO, Concurrency-aware cost paper)
FAQ
How do you calculate GPU cost per million tokens?
Divide the fully allocated hourly cost by effective tokens per hour, then scale to one million: R × 1,000,000 ÷ (tokens per second × 3,600 × utilization). Measure tokens per second at your real concurrency and latency limit, and allocate storage, network, software, and operator time into R. (Source: NVIDIA inference TCO)
How does GPU utilization affect cost per token?
Low utilization spreads the same hourly GPU charge across fewer tokens, increasing unit cost. Higher utilization lowers cost per token until batching or concurrency violates the workload's latency target. Measure effective throughput at the service-level objective instead of using a hardware peak. (Source: NVIDIA inference TCO)
Which cloud GPU is cheapest for LLM inference?
The cheapest hourly instance is not necessarily the cheapest inference option. Compare each GPU's fully allocated hourly rate with measured throughput for the same model, quantization, batch size, context, and latency target. A more expensive GPU can have a lower cost per million tokens when it produces substantially more usable throughput. (Sources: AWS EC2 pricing; NVIDIA inference TCO)
How do batching and concurrency reduce GPU cost per million tokens?
Batching and concurrency keep more of the accelerator occupied, increasing effective tokens per hour and spreading the hourly charge across more output. The benefit stops being useful when queueing or larger batches exceed the application's latency objective, so report both throughput and latency. (Source: NVIDIA inference TCO)
Is it cheaper to self-host an LLM than use an API?
Self-hosting is cheaper only above the break-even utilization for your rate and throughput. Below it, the API wins because someone else carries idle capacity and operations. Calculate R × 1,000,000 ÷ (tokens per second × 3,600 × A), then add migration, failover, and on-call cost before deciding. (Source: AWS EC2 pricing)
What costs besides the GPU hourly price affect LLM inference?
Include CPU and memory attached to the instance, storage, network transfer, idle capacity, replicas, failover, monitoring, software, and operator time. For a fair API comparison, allocate these costs into the effective hourly rate before calculating cost per million tokens. (Source: NVIDIA inference TCO)
Related coverage
Use our LLM VRAM calculator to check model fit before pricing a self-hosted GPU, and see what running a local LLM on 8GB RAM costs when the answer is no cloud GPU at all. To cut the API side of the comparison, see prompt caching cost reduction. To turn a per-token figure into a per-outcome one, see AI agent cost per successful task.
References
- Anthropic pricing - https://platform.claude.com/docs/en/about-claude/pricing
- AWS EC2 pricing - https://aws.amazon.com/ec2/pricing/on-demand/
- Concurrency-aware cost paper - https://arxiv.org/abs/2606.11690
- Google Cloud GPU pricing - https://cloud.google.com/products/compute/pricing
- Google Gemini pricing - https://ai.google.dev/gemini-api/docs/pricing
- Lambda GPU cloud pricing - https://lambda.ai/service/gpu-cloud
- NVIDIA inference TCO - https://developer.nvidia.com/blog/llm-inference-benchmarking-how-much-does-your-llm-inference-cost/
- Runpod pricing - https://www.runpod.io/pricing
