GPU cost per million tokens is the hourly rate divided by what the GPU actually produces: GPU-hour rate × 1,000,000 ÷ (tokens per second × 3,600), adjusted for utilization. A cloud GPU pricing comparison is only half an answer until you make that conversion, and the result is not a production cost until it also carries uptime, storage, software, and operations.
This page does the conversion for ten GPU list prices from Runpod and Lambda, read on 2026-08-29, then works out when a self-hosted GPU beats a hosted API. The same H100 rents for $1.99 per hour on one provider's community tier and $4.29 per hour on another's dedicated cloud. At $2.00 per hour and 120 tokens per second, a deployment breaks even with Claude Opus 5 near 31% utilization and with Claude Sonnet 5 near 77%, and it cannot reach the price of Claude Haiku 4.5 or Gemini 2.5 Flash at any utilization.
The calculator at the top of this page runs the same formulas. It opens on the worked example used throughout this guide ($2.00 per hour, 120 tokens per second, 50% utilization), and you can change the rate, throughput, utilization, overhead, and API price to get cost per million tokens and the break-even utilization for your own deployment. Measure tokens per second under the latency and concurrency your users need before you trust the output; peak benchmark speed is not a business metric until the deployment can sustain it.
GPU cost per million tokens formula
Start with the simplest steady-state equation:
Cost per 1M tokens = hourly GPU cost × 1,000,000 ÷ effective tokens per hour
Effective tokens per hour are tokens per second × 3,600 × utilization. If you include fixed costs, replace the hourly GPU rate with an allocated rate that also covers storage, software, monitoring, support, and depreciation. NVIDIA's inference TCO guidance treats throughput, latency, demand, uptime, and infrastructure as connected inputs.
| Variable | Meaning | Example |
|---|---|---|
R | GPU and allocated infrastructure cost per hour | $2.00 |
T | Measured combined tokens per second | 120 |
U | Sustained utilization | 50% |
C | Cost per million tokens | $9.26 |
The example is arithmetic, not a provider quote. At $2.00, 120 tokens/s, and 50% utilization, the result is $2 × 1,000,000 ÷ (120 × 3,600 × 0.50) = $9.26. At 80% utilization with the same speed and rate, it falls to $5.79.
NVIDIA's method also separates input and output token costs, because output generation often has different performance and pricing behavior. Record the input/output mix instead of reporting one blended speed without context.
Assumptions behind the numbers on this page
Every figure below is a list price or arithmetic on one. Nothing on this page is a benchmark of a specific card, so swap in your own measured throughput before you act on a ranking.
| Input | Value used here | Where it comes from |
|---|---|---|
| GPU hourly rates | $0.33 to $4.29 per hour | Runpod and Lambda list prices, read on 2026-08-29 |
| API token prices | Per-model input and output list prices | Anthropic and Google pricing pages, read on 2026-08-27; Google's page carries a 2026-08-26 update stamp |
| Throughput | 120 tokens per second, held constant for every GPU | Illustrative figure, not a measurement |
| Hourly rate in worked examples | $2.00 | Illustrative round number, not a provider quote |
| Utilization | 100% in the provider table, 10% to 80% in the utilization table, solved for in the break-even table | Planning scenarios |
| Overhead | 15%, in one example only | Planning assumption, not a universal cloud fee |
| Input/output split | 50/50 for blended API prices | Modeling assumption, not a provider term |
| Failed-run share | 20%, in the useful-output example | Stated assumption |
Cloud GPU pricing comparison: current on-demand rates
The rates below are the list prices on each provider's own pricing page. The right-hand column applies the formula above at a held-constant 120 tokens per second, which is 432,000 tokens per hour at full utilization. Holding throughput constant isolates the price variable; it is not a claim that these cards perform alike. An H100 and an RTX A6000 do not produce the same tokens per second, so the real ranking changes once you substitute your own measured rate.
| GPU | Provider tier | $/hour | $/MTok at 120 tok/s, 100% utilization |
|---|---|---|---|
| RTX A6000 48GB | Runpod community | $0.33 | $0.76 |
| L40S 48GB | Runpod community | $0.79 | $1.83 |
| A6000 48GB | Lambda on-demand | $1.09 | $2.52 |
| A100 PCIe 80GB | Runpod community | $1.19 | $2.75 |
| A100 SXM 40GB | Lambda on-demand | $1.99 | $4.61 |
| H100 PCIe 80GB | Runpod community | $1.99 | $4.61 |
| H100 SXM 80GB | Runpod community | $2.69 | $6.23 |
| H100 PCIe 80GB | Runpod secure | $2.89 | $6.69 |
| H100 PCIe 80GB | Lambda on-demand | $3.29 | $7.62 |
| H100 SXM 80GB | Lambda on-demand | $4.29 | $9.93 |
Two things in that table decide more than the GPU choice. First, the same H100 PCIe card spans $1.99 to $3.29 per hour across these tiers, a 65% spread on identical silicon, because a community marketplace, a dedicated secure tier, and a managed AI cloud sell different reliability rather than different hardware. Second, the cheapest row is only the cheapest if the card sustains the throughput your latency target needs: an RTX A6000 at $0.76 per million tokens loses to an H100 at $4.61 the moment the smaller card cannot hold your concurrency. Price the tier and the throughput together.
Cloud GPU cost per hour is only the starting point
Cloud GPU cost per hour varies with region, GPU type, machine shape, operating system, commitment, and billing model. AWS and Google Cloud publish provider-specific tables, but those are rate sources, not throughput benchmarks: they do not tell you how many tokens your model will generate. Record the region, machine type, GPU count, and billing unit beside every copied price, then benchmark the deployment.
The hourly line can also exclude costs that appear elsewhere on the bill: snapshots, load balancers, orchestration, logging, reserved capacity commitments, and idle replicas. For a self-hosted service, include hardware depreciation, power, cooling, rack space, and the operator time needed to keep it available.
| Cost input | Include it when | Common mistake |
|---|---|---|
| GPU instance | Always | Comparing different GPU generations as if they were equivalent |
| CPU, RAM, and host | The model or runtime needs them | Counting only accelerator price |
| Storage and snapshots | Models or logs persist | Ignoring idle and backup storage |
| Network and egress | Responses leave the region or VPC | Treating transfer as free |
| Software and operations | The service runs beyond a test | Leaving monitoring and on-call out |
Cloud GPU cost comparison by utilization
Utilization is the percentage of the paid deployment's available generation capacity that produces useful tokens. It is not the CPU percentage shown in a dashboard. A queue can keep a GPU busy while a latency target is failing, so pair utilization with TTFT, intertoken latency, and request success, as NVIDIA's benchmark guidance does.
The table below holds the GPU rate at $2.00/hour and measured output at 120 tokens/s.
| Utilization | Effective tokens per hour | Cost per million tokens |
|---|---|---|
| 10% | 43,200 | $46.30 |
| 25% | 108,000 | $18.52 |
| 50% | 216,000 | $9.26 |
| 80% | 345,600 | $5.79 |
Add 15% overhead for storage, software, and operations and the 50% example becomes $10.65 per million tokens. Report the result as a range with that assumption visible.
Third-party comparison pages make the hourly-to-token conversion easier to see, but each one bakes in its own model, rate, and throughput. The concurrency-aware methodology paper makes the underlying objection: a single per-token figure hides the workload shape and service constraints that produced it. Use a published comparison to find the terms, then recalculate from a rate and benchmark that match your deployment.
Why concurrency changes GPU cost per token
Concurrency is the number of active requests being served at the same time. Low concurrency can give each request a fast response while leaving the GPU underused. Higher concurrency can improve batching and tokens per second, but it can also increase time to first token and queueing.
NVIDIA recommends measuring TTFT, intertoken latency, tokens per second, requests per second, and concurrency, and its example treats a latency limit as a constraint before selecting the highest-throughput operating point. The arXiv paper makes the same correction to simple per-token pricing: infrastructure cost changes with workload shape, service constraints, and the capacity needed to handle demand. The cheapest point on a throughput chart is irrelevant if it misses your response-time target, so report the concurrency used for every cost per million tokens number.
For a useful benchmark, hold the model, quantization, prompt lengths, output cap, runtime, and sampling settings constant. Run concurrency levels such as 1, 2, 4, 8, and 16. Keep the best point that meets your TTFT and error-rate limits, then calculate unit cost from that point. The serving runtime sets how far concurrency can push that number; the vLLM vs Ollama comparison shows published throughput for both at 1 to 256 concurrent users.
Cloud GPU pricing comparison versus an API
Compare a GPU with an API at the same token mix and quality level. If the API charges $A per million combined tokens and the GPU's fully allocated hourly rate is R, the utilization needed to match the API is:
Break-even utilization = R × 1,000,000 ÷ (tokens per second × 3,600 × A)
Refresh the API prices and the GPU rates together, because either side can move the break-even point.
Provider prices to test the break-even against
The table holds the deployment at $2.00/hour and 120 tokens/s, which is 432,000 tokens per hour at full utilization, and asks what utilization each published API price demands. Prices are the list rates on Anthropic's pricing page and Google's Gemini pricing page. The blended column assumes a 50/50 input-output split.
| Provider and model | Input $/MTok | Output $/MTok | Blended 50/50 $/MTok | Break-even utilization |
|---|---|---|---|---|
| Claude Haiku 4.5 | $1.00 | $5.00 | $3.00 | not reachable (154%) |
| Gemini 2.5 Flash | $0.30 | $2.50 | $1.40 | not reachable (331%) |
| Gemini 3.5 Flash | $1.50 | $9.00 | $5.25 | 88.2% |
| Gemini 2.5 Pro | $1.25 | $10.00 | $5.62 | 82.3% |
| Claude Sonnet 5 | $2.00 | $10.00 | $6.00 | 77.2% |
| Claude Opus 5 | $5.00 | $25.00 | $15.00 | 30.9% |
Two rows exceed 100%, which means this deployment cannot reach those prices at any utilization: a small self-hosted instance does not beat a cheap hosted model on price alone. The frontier rows are the beatable ones. Against Claude Opus 5 the same hardware breaks even near 31% utilization, and against Claude Sonnet 5 it needs 77%. That result excludes migration cost and failover, so add those before choosing self-hosting.
One caveat changes the comparison and appears in no price table. Anthropic states that Claude 4.7 and later models use a newer tokenizer that produces roughly 30% more tokens for the same text, while Sonnet 4.6 and earlier use the previous one. A per-token price is therefore not comparable across tokenizers: the same document costs more on the newer models at an identical rate. Prices also move on schedule. Google lists Gemini 3.7 and 3.6 Flash at $0.75 input and $3.75 output through 2026-12-31, then $1.50 and $7.50. Compare on cost per unit of finished work, and rerun the equation against both pages before quoting a break-even number.
| Deployment choice | Usually fits | Decision test |
|---|---|---|
| API | Variable or uncertain demand | Compare current API price with no idle commitment |
| On-demand GPU | Stable bursts and a known model | Test utilization and scale-up delay |
| Reserved or committed GPU | High, predictable demand | Include commitment risk and capacity floor |
| Local or owned GPU | Data locality and steady use | Add depreciation, power, support, and failure recovery |
The break-even number is a threshold, not a verdict. An API may win at low utilization because someone else carries capacity and operations. A GPU may win at steady high utilization when the team can operate it safely. Before you accept the API price on the right side of the equation, check whether prompt caching cost reduction lowers it, because a cached prefix moves the break-even point against self-hosting.
GPU inference cost calculator inputs
A GPU inference cost calculator needs more than a GPU name and an hourly price. Capture these inputs in the same run that produces your performance number, so the unit cost stays tied to a reproducible serving configuration rather than a generic hardware label:
- Model name, version, quantization, context length, and input/output token mix.
- Runtime and serving configuration, including batching and memory settings.
- GPU type, count, region, hourly rate, and any commitment term.
- Concurrency, TTFT, intertoken latency, tokens per second, requests per second, and error rate.
- Sustained utilization, uptime target, storage, network, monitoring, and operator allocation.
Raw tokens versus useful output
Cost per million generated tokens is a serving metric, not a measure of completed work. If 20% of generated tokens belong to failed or rejected runs, a $10 cost per million raw tokens becomes $12.50 per million accepted tokens under that stated assumption: divide $10 by 0.8. The arithmetic does not account for different prompt/output mixes or review time. Keep the raw throughput figure for infrastructure comparisons, then use cost per successful agent task when the buyer pays for an outcome.
Related coverage
- Model fit: the LLM VRAM calculator checks whether a model fits before you price a self-hosted GPU.
- No cloud GPU at all: running a local LLM on 8GB RAM covers what that costs.
- The API side: prompt caching cost reduction shows how to cut it.
- Per outcome: AI agent cost per successful task turns a per-token figure into a per-task one.
References
- Anthropic pricing - https://platform.claude.com/docs/en/about-claude/pricing
- AWS EC2 pricing - https://aws.amazon.com/ec2/pricing/on-demand/
- Concurrency-aware cost paper - https://arxiv.org/abs/2606.11690
- Google Cloud GPU pricing - https://cloud.google.com/products/compute/pricing
- Google Gemini pricing - https://ai.google.dev/gemini-api/docs/pricing
- Lambda GPU cloud pricing - https://lambda.ai/service/gpu-cloud
- NVIDIA inference TCO - https://developer.nvidia.com/blog/llm-inference-benchmarking-how-much-does-your-llm-inference-cost/
- Runpod pricing - https://www.runpod.io/pricing
