GPU Cost per Million Tokens: The Utilization Math

GPU cost per million tokens is the hourly cost of an inference deployment multiplied by 1,000,000, divided by the tokens it produces per hour. The basic equation is GPU-hour rate × 1,000,000 ÷ (tokens per second × 3,600). Add utilization, uptime, storage, software, and operations before calling the result a production cost.

That distinction matters because a GPU-hour price is only an input. A card that costs $2 per hour can be cheaper or more expensive than an API depending on throughput, concurrency, model size, quantization, context length, and idle time. This guide gives you a cloud GPU cost comparison method that you can recalculate with a provider's current rate and your measured workload.

The practical answer: measure tokens per second under the latency and concurrency your users need, then divide the fully allocated hourly cost by effective tokens per hour. Peak benchmark speed is not a business metric until the deployment can sustain it.

GPU cost per million tokens formula

Start with the simplest steady-state equation:

Cost per 1M tokens = hourly GPU cost × 1,000,000 ÷ effective tokens per hour

Effective tokens per hour are tokens per second × 3,600 × utilization. If you include fixed costs, replace the hourly GPU rate with an allocated rate that also covers storage, software, monitoring, support, and depreciation. NVIDIA's TCO guidance treats throughput, latency, demand, uptime, and infrastructure as connected inputs. (Source: NVIDIA inference TCO)

The example is arithmetic, not a provider quote. At $2.00, 120 tokens/s, and 50% utilization, the result is $2 × 1,000,000 ÷ (120 × 3,600 × 0.50) = $9.26. If the workload produces 80% utilization with the same speed and rate, the result falls to $5.79.

NVIDIA's inference TCO method also separates input and output token costs because output generation often has different performance and pricing behavior. Record the input/output mix instead of reporting one blended speed without context. (Source: NVIDIA inference TCO)

Cloud GPU cost per hour is only the starting point

Cloud GPU cost per hour varies with region, GPU type, machine shape, operating system, commitment, and billing model. AWS and Google Cloud publish provider-specific tables, but those tables do not tell you how many tokens your model will generate. Copy the rate that matches your region and configuration, then benchmark the deployment.

The hourly line can also exclude costs that appear elsewhere on the bill. Add attached storage, snapshots, network egress, load balancers, orchestration, logging, reserved capacity commitments, and idle replicas. For a self-hosted service, include hardware depreciation, power, cooling, rack space, and the operator time needed to keep it available.

AWS's on-demand page and Google Cloud's GPU pricing pages are rate sources, not throughput benchmarks. Record the region, machine type, GPU count, and billing unit beside every copied price. (Source: AWS EC2 pricing, Google Cloud GPU pricing)

Cloud GPU cost comparison by utilization

Utilization is the percentage of the paid deployment's available generation capacity that produces useful tokens. It is not the CPU percentage shown in a dashboard. A queue can keep a GPU busy while a latency target is failing, so pair utilization with TTFT, intertoken latency, and request success. NVIDIA's benchmark guidance uses these latency and throughput measures together rather than treating one peak speed as the answer. (Source: NVIDIA inference TCO)

The table below holds the GPU rate at $2.00/hour and measured output at 120 tokens/s. It shows why a cloud GPU pricing comparison without utilization can mislead a buyer.

Add 15% overhead for storage, software, and operations and the 50% example becomes $10.65 per million tokens. That surcharge is an explicit planning assumption, not a universal cloud fee. The useful output is a range with the assumption visible.

Third-party comparison pages make the hourly-to-token conversion easier to see, but each one bakes in its own model, rate, and throughput inputs. The concurrency-aware methodology paper makes the underlying objection: a single per-token figure hides the workload shape and service constraints that produced it. Use a published comparison to find the terms, then recalculate from a rate and benchmark that match your deployment. (Source: Concurrency-aware cost paper)

Why concurrency changes GPU cost per token

Concurrency is the number of active requests being served at the same time. Low concurrency can give each request a fast response while leaving the GPU underused. Higher concurrency can improve batching and tokens per second, but it can also increase time to first token and queueing.

NVIDIA recommends measuring TTFT, intertoken latency, tokens per second, requests per second, and concurrency. Its example treats a latency limit as a constraint before selecting the highest-throughput operating point. The cheapest point on a throughput chart is irrelevant if it misses your response-time target. (Source: NVIDIA inference TCO)

The arXiv concurrency-aware methodology makes the same correction to simple per-token pricing. Infrastructure cost changes with workload shape, service constraints, and the capacity needed to handle demand. Report the concurrency used for every GPU cost per million tokens number. (Source: Concurrency-aware cost paper)

For a useful benchmark, hold the model, quantization, prompt lengths, output cap, runtime, and sampling settings constant. Run concurrency levels such as 1, 2, 4, 8, and 16. Keep the best point that meets your TTFT and error-rate limits, then calculate unit cost from that point.

Cloud GPU pricing comparison versus an API

Compare a GPU with an API at the same token mix and quality level. If the API charges $A per million combined tokens and the GPU's fully allocated hourly rate is R, the utilization needed to match the API is shown by the following equation. Provider price tables and GPU rates should be refreshed together because either side can change the break-even point. (Source: AWS EC2 pricing)

Break-even utilization = R × 1,000,000 ÷ (tokens per second × 3,600 × A)

Provider prices to test the break-even against

The table below holds the deployment at $2.00/hour and 120 tokens/s, which is 432,000 tokens per hour at full utilization, and asks what utilization each published API price demands. Prices are the list rates on the providers' own pages, read on 2026-08-27; Google's page carries a 2026-08-26 update stamp. Blended rates assume a 50/50 input-output split, which is a modeling assumption, not a provider term. (Source: Anthropic pricing, Google Gemini pricing)

Two rows exceed 100%, which means this deployment cannot reach those prices at any utilization: a small self-hosted instance does not beat a cheap hosted model on price alone. The frontier rows are the beatable ones. Against Claude Opus 5 the same hardware breaks even near 31% utilization, and against Claude Sonnet 5 it needs 77%. That result excludes migration cost and failover. Add those costs before choosing self-hosting.

One caveat changes the comparison and appears in no price table. Anthropic states that Claude 4.7 and later models use a newer tokenizer producing roughly 30% more tokens for the same text, while Sonnet 4.6 and earlier use the previous one, so a per-token price is not comparable across tokenizers: the same document costs more on the newer models at an identical rate. Prices also move on schedule; Google lists Gemini 3.7 and 3.6 Flash at $0.75 input and $3.75 output through 2026-12-31, then $1.50 and $7.50. Compare on cost per unit of finished work, and rerun the equation against both pages before quoting a break-even number. (Source: Anthropic pricing, Google Gemini pricing)

The break-even number is a threshold, not a verdict. An API may win at low utilization because someone else carries capacity and operations. A GPU may win at steady high utilization when the team can operate it safely. Before you accept the API price on the right side of that equation, check whether prompt caching cost reduction lowers it, because a cached prefix moves the break-even point against self-hosting.

GPU inference cost calculator inputs

An honest GPU inference cost calculator needs more than GPU name and hourly price. Capture these inputs in the same run that produces your performance number. This keeps the unit-cost result tied to a reproducible serving configuration rather than a generic hardware label. (Source: Concurrency-aware cost paper)

1. Model name, version, quantization, context length, and input/output token mix.
2. Runtime and serving configuration, including batching and memory settings.
3. GPU type, count, region, hourly rate, and any commitment term.
4. Concurrency, TTFT, intertoken latency, tokens per second, requests per second, and error rate.
5. Sustained utilization, uptime target, storage, network, monitoring, and operator allocation.

NVIDIA's GenAI-Perf guidance is useful because it turns throughput and latency into infrastructure sizing inputs. The paper's contribution is equally practical: an infrastructure estimate should reflect the service conditions that make the deployment acceptable, not a single peak number. (Source: NVIDIA inference TCO, Concurrency-aware cost paper)

FAQ

How much does a 1 million token cost?

One million tokens has no fixed cost. A hosted API uses its input and output rates. A GPU deployment uses hourly cost divided by effective tokens per hour, including utilization and overhead. State the model, token mix, throughput, region, and capacity assumptions before comparing two prices. NVIDIA's methodology supports reporting throughput and latency conditions beside a cost result. (Source: NVIDIA inference TCO)

How big is 1,000,000 tokens?

One million tokens is a billing quantity, not a fixed document size. The number of words or pages depends on the tokenizer, language, code density, and formatting. For infrastructure math, use the token counts reported by the selected provider or tokenizer rather than converting from pages with a blanket rule.

Is 1 million tokens a lot?

Whether one million tokens is a lot depends on the workload. It may represent many short requests or a few long-context generations. For a GPU cost comparison, the useful questions are how quickly those tokens arrive, how much concurrency they require, and whether the deployment must stay online between requests.

How much do 1000 tokens cost?

The cost of 1,000 tokens is the relevant per-million-token price divided by 1,000, with input and output treated separately when the provider uses different rates. For a GPU, divide effective hourly cost by effective tokens per hour, then multiply by 1,000.

Is it cheaper to self-host an LLM than use an API?

Self-hosting is cheaper only above the break-even utilization for your rate and throughput. Below it, the API wins because someone else carries idle capacity and operations. Calculate R × 1,000,000 ÷ (tokens per second × 3,600 × A), then add migration, failover, and on-call cost before deciding. (Source: AWS EC2 pricing)

How do you calculate GPU cost per million tokens?

Divide the fully allocated hourly cost by effective tokens per hour, then scale to one million: R × 1,000,000 ÷ (tokens per second × 3,600 × utilization). Measure tokens per second at your real concurrency and latency limit, and allocate storage, network, software, and operator time into R. (Source: NVIDIA inference TCO)

Related coverage

Use our LLM VRAM calculator to check model fit before pricing a self-hosted GPU. To cut the API side of the comparison, see prompt caching cost reduction. To turn a per-token figure into a per-outcome one, see AI agent cost per successful task.

References

• Anthropic pricing - https://platform.claude.com/docs/en/about-claude/pricing
• AWS EC2 pricing - https://aws.amazon.com/ec2/pricing/on-demand/
• Concurrency-aware cost paper - https://arxiv.org/abs/2606.11690
• Google Cloud GPU pricing - https://cloud.google.com/products/compute/pricing
• Google Gemini pricing - https://ai.google.dev/gemini-api/docs/pricing
• NVIDIA inference TCO - https://developer.nvidia.com/blog/llm-inference-benchmarking-how-much-does-your-llm-inference-cost/