Inference Cost Economics: Tokens per Dollar, GPU-Hours, and the Real Math of LLM Serving

Everyone talks about LLM inference performance in tokens per second. Nobody talks about the number that actually determines whether your product is viable: tokens per dollar. A system that generates 1,000 tokens per second at USD 10/hour costs USD 2.78 per million tokens. A system that generates 200 tokens per second at USD 1/hour costs USD 1.39 per million tokens. The slower system is cheaper by half.

This disconnect between speed and cost is pervasive in the LLM serving world. Engineers optimize for throughput benchmarks while finance teams stare at cloud bills. The gap between “fast” and “cheap” grows wider as you add batching, quantization, and hardware choices to the equation. Understanding the real math of inference economics is essential for anyone deploying LLMs at scale.

This post builds the complete cost model from first principles: the fundamental equation linking hardware cost to token cost, GPU pricing across clouds and purchase models, how each major optimization technique affects unit economics, the asymmetry between input and output token costs, the throughput-latency tradeoff frontier, when to self-host vs. use APIs, distillation as a cost lever, and where costs are heading with next-generation hardware.

---

1. The Fundamental Equation

Cost Per Token

Every inference cost ultimately reduces to one equation:

$\text{Cost per token} = \frac{\text{Cost per GPU-hour}}{\text{Tokens per GPU-hour}}$

This is a fraction with two levers: reduce the numerator (cheaper hardware) or increase the denominator (more tokens per hour from the same hardware). Every optimization technique in the LLM inference stack attacks one or both of these.

Expanding the denominator:

$\text{Tokens per GPU-hour} = \text{Tokens/sec/GPU} \times 3600$

And tokens per second depends on the model, quantization, batch size, hardware, and sequence length:

$\text{Tokens/sec} = f(\text{model\_size}, \text{quant}, \text{batch\_size}, \text{hardware}, \text{seq\_len})$

There is no simple closed-form expression for this function — it depends on whether you are in the compute-bound or memory-bandwidth-bound regime, the efficiency of the serving software, and the workload characteristics. But we can derive useful approximations.

The Bandwidth-Bound Approximation

For decode (single-token generation), the throughput is approximately:

$\text{Decode tok/s} \approx \frac{\text{Memory BW (GB/s)}}{\text{Model size (GB)} / \text{batch\_size}}$

Wait — that is not quite right. More precisely, for batch size $B$:

$\text{Decode tok/s} \approx \frac{B \times \text{Memory BW}}{\text{Model Size} + B \times \text{KV cache per token} \times \text{seq\_len}}$

For small batch sizes where the model weight read dominates, this simplifies to:

$\text{Decode tok/s} \approx B \times \frac{\text{Memory BW}}{\text{Model Size}}$

The throughput scales linearly with batch size until you hit the compute bound (tensor core saturation) or memory capacity limit (KV cache fills up VRAM).

The Compute-Bound Approximation

For prefill (processing input tokens), at large batch sizes the throughput is approximately:

$\text{Prefill tok/s} \approx \frac{\text{FLOPS}}{2 \times N_{\text{params}}}$

where $N_{\text{params}}$ is the model parameter count and the factor of 2 accounts for the multiply-add per parameter. This is independent of batch size once the GEMMs are large enough to saturate the tensor cores (typically at $M \geq 128$ for the batch dimension).

For Llama 70B on an H100 (989 TFLOPS FP16):

$\text{Prefill tok/s} \approx \frac{989 \times 10^{12}}{2 \times 70 \times 10^9} \approx 7{,}064 \text{ tok/s}$

This is the theoretical peak. In practice, FlashAttention overhead, memory access patterns, and kernel launch latency reduce this to ~4,000-5,000 tok/s.

---

2. GPU Economics: The Hardware Cost

Cloud GPU Pricing

GPU pricing varies dramatically across providers, instance types, and commitment levels. Here is the landscape as of early 2025:

The pricing range for the same GPU (H100) spans from USD 2.23/hr (CoreWeave) to USD 12.29/hr (AWS on-demand). That is a 5.5x difference for identical hardware. The cloud provider choice alone can cut costs by 80%.

On-Premises GPU Economics

For sustained workloads, purchasing GPUs can be dramatically cheaper than cloud rental:

$\text{Effective hourly cost} = \frac{\text{Purchase price} + \text{3-year opex}}{\text{Hours in 3 years} \times \text{utilization}}$

For an H100:

• Purchase price: ~USD 30,000
• 3-year electricity (700W, USD 0.10/kWh): USD 18,400
• 3-year hosting/cooling: ~USD 5,000
• Total 3-year cost: ~USD 53,400
• Hours in 3 years at 90% utilization: 23,652

$\text{Effective cost} = \frac{53{,}400}{23{,}652} \approx \USD 2.26/\text{hr}$

This is comparable to the cheapest cloud providers — and you own the hardware. But on-premises requires upfront capital, operations expertise, and bears the risk of hardware depreciation (the H100 may be worth much less in 3 years when Blackwell-successor chips are available).

---

3. Tokens Per Second Per Dollar: The Optimization Stack

How Each Optimization Affects Unit Economics

The cost per token depends on how many tokens per second you extract from each dollar of GPU cost. Here is how each major optimization technique contributes:

Quantization (2-4x cost reduction). Reducing model precision from FP16 to INT4 roughly halves the model size, doubling the effective memory bandwidth and (in the bandwidth-bound regime) doubling throughput. On hardware with INT4/INT8 tensor cores (H100, Ada Lovelace), quantization also increases compute throughput.

$\text{Speedup from quantization} \approx \frac{\text{Bits}_{\text{original}}}{\text{Bits}_{\text{quantized}}} = \frac{16}{4} = 4\text{x (theoretical maximum)}$

In practice, quantization overhead (dequantization, scale multiplication) and mixed-precision attention reduce this to 2-3x.

Batching (5-10x cost reduction). As discussed in Section 1, decode throughput scales linearly with batch size until compute saturation. Going from batch size 1 to batch size 32 can increase throughput by 20-30x while only increasing per-request latency by 2-3x (from KV cache memory pressure and scheduling overhead).

FlashAttention (1.3-2x cost reduction). FlashAttention reduces attention memory and compute overhead, enabling larger batch sizes (more KV cache fits in VRAM) and faster prefill. The direct speedup is modest (1.3-1.5x), but the indirect speedup from enabling larger batches can be 2x+.

Speculative decoding (2-3x cost reduction for latency). Speculative decoding reduces wall-clock time per token by verifying multiple draft tokens at once. However, it does not improve throughput (total tokens/sec across all requests) — it improves latency (time per token for individual requests). The cost benefit comes from enabling lower per-request latency at the same throughput, allowing you to meet stricter SLOs without overprovisioning.

The fully optimized stack is 72x cheaper per token than the naive baseline. The single biggest lever is batching: going from batch size 1 to batch size 64 accounts for a 23x improvement. Quantization provides another 2.3x. FlashAttention and PagedAttention contribute the remaining efficiency gains.

---

4. Prefill vs. Decode Cost Asymmetry

Why Input Tokens Are Cheaper

If you have used a commercial LLM API, you have noticed that input tokens are priced lower than output tokens — typically 3-10x lower:

The pricing difference reflects the fundamental cost asymmetry between prefill and decode:

Prefill (processing input tokens) is compute-efficient. All input tokens are processed in parallel via large matrix multiplications. On an H100, Llama 70B prefill achieves ~4,000 tokens/sec — the tensor cores are well-utilized. The cost per input token is:

$\text{Prefill cost/token} = \frac{\USD 2.49/\text{hr}}{4{,}000 \text{ tok/s} \times 3{,}600} = \USD 0.000000173 \approx \USD 0.17/\text{M tokens}$

Decode (generating output tokens) is bandwidth-inefficient. Each output token requires reading the entire model from memory — the same weight traffic as processing hundreds of input tokens, but producing only one token. At batch size 32, Llama 70B decode achieves ~1,000 tokens/sec:

$\text{Decode cost/token} = \frac{\USD 2.49/\text{hr}}{1{,}000 \text{ tok/s} \times 3{,}600} = \USD 0.000000692 \approx \USD 0.69/\text{M tokens}$

The decode cost is 4x higher than prefill, explaining the typical API pricing ratio.

Implications for Prompt Design

This cost asymmetry has practical implications:

1. Long prompts are relatively cheap. A 10,000-token system prompt costs roughly USD 0.0017 per request (at USD 0.17/M tokens for prefill). This is negligible compared to generating even 500 output tokens (USD 0.00035 at USD 0.69/M tokens).

2. Few-shot examples in prompts are cost-effective. Adding 10 examples at 200 tokens each (2,000 extra input tokens) costs ~USD 0.00034 but can significantly reduce the output length needed (the model gets the format right on the first try instead of generating verbose explanations).

3. Reasoning models invert the ratio. A reasoning model generating 20,000 thinking tokens per query makes output tokens the dominant cost by 100x+. The input cost becomes completely negligible.

For summarization tasks (long input, short output), input tokens dominate the cost. For code generation and reasoning (moderate input, long output), output tokens dominate. Optimizing the expensive phase is what matters — there is no point in optimizing prefill for a reasoning workload.

---

5. The Batch Size Lever: Throughput vs. Latency

The Pareto Frontier

Batch size creates a fundamental tradeoff between throughput (tokens/sec for the whole system) and latency (time per token for each individual request). Larger batches improve throughput but increase latency because:

1. Memory contention. More concurrent requests mean more KV cache in VRAM, reducing the memory available for other operations.
2. Compute sharing. At very large batch sizes, the decode phase transitions from memory-bandwidth-bound to compute-bound, and adding more requests no longer increases throughput.
3. Scheduling overhead. More requests mean more scheduling decisions, more preemption events, and more KV cache management overhead.

The Pareto frontier is clearly visible:

• Batch 1-16: Throughput scales nearly linearly. Latency barely increases. This is the “free lunch” regime — you are simply utilizing idle hardware capacity.
• Batch 16-64: Throughput still scales well but latency starts climbing. The GPU is becoming well-utilized.
• Batch 64-128: Throughput gains slow. Latency increases substantially. KV cache pressure forces some requests to queue.
• Batch 128-256: Throughput plateaus. Latency degrades significantly. Diminishing returns.

Choosing the Operating Point

The optimal batch size depends on your SLO:

• Real-time chat (latency target: under 50ms/tok): Batch size 32-64. Cost ~USD 0.45-0.78/M tokens.
• Near-real-time (latency target: under 100ms/tok): Batch size 128. Cost ~USD 0.33/M tokens.
• Batch processing (no latency constraint): Maximum batch size. Cost ~USD 0.28/M tokens.

The cost difference between real-time and batch processing is “only” 1.6-2.8x. Many operators choose the real-time operating point even for batch workloads because the throughput is still high enough and the system can also serve interactive requests.

---

6. Build vs. Buy: Self-Hosting vs. API

The Break-Even Analysis

The most common economic question in LLM deployment is: should we self-host (run our own GPU infrastructure) or use an API (pay per token)?

The answer depends on volume. APIs have zero fixed cost but high marginal cost per token. Self-hosting has high fixed cost (GPU rental) but low marginal cost per token.

$\text{API cost} = V \times P_{\text{api}}$

$\text{Self-host cost} = C_{\text{gpu}} \times H + C_{\text{ops}}$

where $V$ is token volume, $P_{\text{api}}$ is API price per token, $C_{\text{gpu}}$ is GPU hourly cost, $H$ is hours of GPU usage, and $C_{\text{ops}}$ is operations overhead (engineering time, monitoring, etc.).

The break-even point is where these are equal:

$V_{\text{break-even}} = \frac{C_{\text{gpu}} \times H + C_{\text{ops}}}{P_{\text{api}} - P_{\text{self-host}}}$

Let us compute this for a concrete scenario: Llama 70B class model, comparing GPT-4o API vs. self-hosted on an H100.

API cost (GPT-4o): USD 10.00/M output tokens, USD 2.50/M input tokens. Assuming 50/50 input/output: USD 6.25/M tokens average.

Self-host cost (H100 on Lambda): USD 2.49/hr, achieving 1,500 tok/s at batch 64. Cost per million tokens: USD 2.49 / (1,500 x 3.6) = USD 0.46/M tokens. Plus engineering overhead: ~USD 5,000/month for a part-time ML engineer.

The break-even is roughly at 3-5 million tokens per day. Below that, the API wins because the GPU sits partially idle and you are paying for engineering overhead that exceeds the API cost. Above that, self-hosting wins decisively — at 50M tokens/day, self-hosting saves USD 273,550/month.

Hidden Costs of Self-Hosting

The GPU hourly rate is only part of the self-hosting cost. Hidden costs include:

1. Model optimization engineering. Getting from naive inference to optimized (quantized, batched, tuned) inference requires significant ML engineering effort. Budget 2-4 engineer-weeks for initial setup.

2. Monitoring and alerting. GPU utilization, latency percentiles, error rates, KV cache pressure, token throughput — all need dashboards and alerts. This is ongoing operational work.

3. Scaling and load balancing. As traffic grows, you need auto-scaling, request routing, and capacity planning. Cloud APIs handle this transparently.

4. Model updates. When a new model version is released, self-hosting requires re-downloading, re-quantizing, re-benchmarking, and re-deploying. APIs update automatically.

5. GPU availability risk. Cloud GPU capacity is not guaranteed. Spot instances can be preempted. On-demand instances may be unavailable during high-demand periods. APIs abstract away this risk.

When APIs Win Even at High Volume

There are scenarios where APIs are worth the premium even at volumes above the break-even:

• Multi-model deployments. If you use 5 different models for different tasks, self-hosting requires 5 separate GPU deployments. APIs let you switch between models per request.
• Rapid iteration. During product development, you may switch models frequently. Self-hosting each candidate model is expensive; API calls let you A/B test instantly.
• Compliance requirements. Some API providers offer SOC 2, HIPAA, and PCI-DSS compliance out of the box. Achieving the same compliance for self-hosted infrastructure is expensive.
• Burst capacity. If your workload has sharp peaks (10x normal traffic for 2 hours per day), APIs absorb the burst without provisioning idle GPUs for the remaining 22 hours. Self-hosting for peak capacity means paying for GPUs that sit idle 90% of the time.

The Hybrid Approach

The most sophisticated operators use a hybrid strategy: self-host a baseline capacity that handles steady-state traffic, and overflow to APIs during peaks. This captures the cost benefit of self-hosting for predictable volume while avoiding overprovisioning for spikes.

For example, if steady-state is 10M tokens/day and peaks reach 40M tokens/day:

• Self-host 1 H100 (handles ~13M tok/day at batch 64): USD 1,800/month
• Overflow ~5M tok/day average to API during peaks: ~USD 940/month (at USD 6.25/M blended)
• Total: USD 2,740/month

Compared to:

• Pure self-host for peak (4 H100s): USD 7,200/month (3 GPUs idle most of the time)
• Pure API: USD 18,750/month

The hybrid approach saves 62% vs. pure self-host and 85% vs. pure API for this workload pattern.

---

7. The Distillation Option

Smaller Models at a Fraction of the Cost

Model distillation — training a smaller model to mimic a larger one — is the most underappreciated cost optimization in LLM serving. A distilled 7B model that achieves 85% of a 70B model’s quality runs at 10x the throughput on the same hardware, reducing cost per token by 10x.

The economics are compelling:

The distilled 13B model at 4-bit quantization achieves 85% of the 70B model’s quality at USD 0.07/M tokens — 6.6x cheaper than the 70B model and comparable quality for many tasks. For applications where “good enough” quality suffices (first-draft generation, simple Q&A, classification, extraction), this is a massive cost saving.

When Distillation Works

Distillation works best when:

1. The task is well-defined. Distilling for “general chat” preserves less quality than distilling for “SQL generation” or “medical triage classification.” Specialized distillation retains 90-95% of teacher quality.

2. The output format is constrained. JSON extraction, classification, structured summarization — these tasks have well-defined output spaces where smaller models can match larger ones closely.

3. The input distribution is known. Distillation on data similar to production inputs works much better than distillation on generic data.

The Cascade Pattern

A sophisticated deployment uses multiple model tiers:

1. Tier 1 (cheap, fast): Distilled 1.5B model handles 60% of requests — simple queries, classification, extraction. Cost: USD 0.02/M tokens.
2. Tier 2 (moderate): Distilled 7B model handles 30% of requests — moderate complexity, basic reasoning. Cost: USD 0.05/M tokens.
3. Tier 3 (expensive, powerful): Full 70B model handles 10% of requests — complex reasoning, creative tasks. Cost: USD 0.46/M tokens.

A confidence-based router sends each request to the cheapest tier that can handle it. If the tier 1 model’s confidence is below a threshold, the request escalates to tier 2, and so on.

The blended cost: $0.02 \times 0.6 + 0.05 \times 0.3 + 0.46 \times 0.1 = 0.012 + 0.015 + 0.046 = 0.073$ USD per million tokens.

This is 6.3x cheaper than using the 70B model for everything, with minimal quality degradation (the hard queries still go to the best model).

---

8. Cost Projections: Where Prices Are Heading

Hardware Trajectory

The cost of inference is driven by hardware economics, and the trajectory is strongly downward:

NVIDIA Blackwell (B100/B200, 2024-2025). Blackwell doubles HBM bandwidth over Hopper (8 TB/s vs. 3.35 TB/s for B200 vs. H100) and adds FP4 tensor cores. The combination yields ~3-4x better inference throughput per GPU for decode-bound workloads. At similar pricing to H100 (USD 2-3/hr), this translates to 3-4x lower cost per token.

FP4 quantization. Blackwell’s native FP4 support enables 4-bit inference without the quality loss of integer quantization. Early benchmarks show FP4 achieving quality within 1% of FP8, at half the memory and double the compute throughput. This is a 2x improvement over the INT4/FP8 that H100 supports.

AMD MI300X. AMD’s MI300X offers 192 GB HBM3 (2.4x the H100’s 80 GB) at competitive pricing (USD 1.50-2.50/hr). The larger memory enables bigger batch sizes without KV cache pressure, improving throughput by 30-50% for memory-constrained workloads. AMD’s software ecosystem (ROCm) is maturing but still lags CUDA.

Algorithmic Trajectory

Hardware improvements compound with algorithmic advances:

Mixture of Experts (MoE). Models like Mixtral 8x7B and DeepSeek V3 (671B total, ~37B active) achieve quality comparable to dense models of similar “active” size but with much faster inference (only the active parameters are read from memory). MoE reduces the effective model size by 3-8x for inference purposes.

Better attention mechanisms. Multi-Query Attention, Grouped-Query Attention, and sliding window attention all reduce KV cache size, enabling larger batch sizes. Future architectures may reduce attention cost further.

Structured pruning. Removing entire attention heads or FFN dimensions that contribute minimally to output quality. This reduces model size without the precision loss of quantization.

Speculative decoding improvements. As draft model quality improves and self-speculative architectures (Medusa, EAGLE) mature, speculative decoding will reduce per-token latency by 2-4x with minimal throughput overhead. This does not directly reduce cost per token in throughput-optimized deployments, but it allows meeting stricter latency SLOs without overprovisioning — an indirect cost saving.

KV cache compression. Techniques like quantized KV cache (FP8 or INT4 for cached keys and values), attention sink pruning (H2O), and sliding window eviction can reduce KV cache memory by 2-4x. This enables larger batch sizes on the same hardware, directly improving throughput and reducing cost per token.

The combined effect of hardware + algorithmic improvements suggests that the cost of a Llama-70B-quality inference will drop by roughly 10x every 2 years. By 2027, the cost per million tokens for a frontier-quality model may be under USD 0.02 — approaching the cost of a web search.

The Price Elasticity of LLM Usage

Understanding cost trajectories matters because LLM usage is highly price-elastic. Empirical data from API providers suggests that a 50% price reduction leads to a 3-4x increase in token volume. This is because cheaper tokens unlock use cases that were previously uneconomical:

• At USD 10/M tokens: only high-value, human-in-the-loop tasks justify LLM usage.
• At USD 1/M tokens: automated pipelines (RAG, extraction, classification) become viable.
• At USD 0.10/M tokens: agent loops (10-50 LLM calls per task) become affordable.
• At USD 0.01/M tokens: continuous LLM processing (every email, every document, every message) becomes practical.

Each 10x cost reduction opens an entirely new tier of applications, each consuming 10-100x more tokens than the previous tier. This is why total inference spending is growing even as unit costs fall — the demand curve is steep.

---

9. Putting It All Together: A Decision Framework

Step 1: Estimate Your Volume

• Under 100K tokens/day: Use APIs. Do not think about infrastructure.
• 100K-5M tokens/day: Use APIs but optimize prompts. Consider distillation for specific high-volume tasks.
• 5M-50M tokens/day: Self-hosting likely wins. Start with managed GPU providers (Lambda, CoreWeave, RunPod).
• Over 50M tokens/day: Definitely self-host. Consider on-premises for sustained workloads.

Step 2: Choose Your Quality-Cost Point

• Maximum quality needed: Use the largest model available. Accept the cost.
• “Good enough” quality for a defined task: Distill a 7B-13B model. 5-15x cheaper.
• Cascade: Route by difficulty. Best quality-cost tradeoff for mixed workloads.

Step 3: Optimize the Serving Stack

For self-hosting, the priority order of optimizations:

1. Continuous batching (5-10x improvement). Use vLLM, SGLang, or TensorRT-LLM — not naive batch inference.
2. Quantization (2-3x improvement). AWQ or GPTQ for GPU, GGUF for CPU. Start with INT4.
3. Model selection (2-10x improvement). Use the smallest model that meets your quality bar. A distilled 13B is better economics than an undistilled 70B for most tasks.
4. Hardware selection (2-5x improvement). Choose the right GPU for your workload. H100 for throughput, A100 for cost-efficiency, Apple Silicon for edge.
5. Attention optimization (1.3-2x improvement). FlashAttention, PagedAttention, and KV cache compression.

Step 4: Monitor and Iterate

Track these metrics continuously:

• Cost per million tokens (the bottom line)
• GPU utilization (aim for over 70% during serving hours)
• Latency percentiles (P50, P95, P99 — ensure SLOs are met)
• Token volume (watch for growth that might cross a break-even threshold)
• Quality metrics (accuracy on evaluation set, user satisfaction — ensure optimizations have not degraded quality)

The gap between the most expensive strategy (USD 15/M tokens for frontier API) and the cheapest (USD 0.04/M tokens for self-hosted distilled model) is 375x. This is not a rounding error — it is the difference between a viable product and an unaffordable one. Understanding inference economics is not optional for anyone building on LLMs.

Step 5: Plan for Cost Evolution

The cost landscape changes rapidly. Decisions that are optimal today may be suboptimal in 6 months. Build your architecture for flexibility:

• Abstract the model layer. Use an internal API that can route to self-hosted or external models. When prices change or new models launch, you can switch without modifying application code.
• Re-evaluate quarterly. GPU pricing, model quality, and API pricing all shift frequently. A quarterly review of your cost model ensures you capture savings.
• Avoid long-term GPU commitments unless volume is stable. One-year reserved instances save 30-40% but lock you in. If a new GPU generation launches mid-commitment, you are stuck on outdated hardware. Prefer 3-month commitments or spot instances for variable workloads.
• Track cost per useful token. Not all generated tokens are useful. Reasoning traces, rejected speculative tokens, and retried generations all cost money but produce no user-facing value. The metric that matters is cost per useful output token delivered to the user.

---

10. Conclusion

The economics of LLM inference are dominated by a few key relationships:

1. Batching is king. The single most impactful optimization, providing 5-30x cost reduction by amortizing the memory bandwidth cost of weight loading across multiple requests.

2. Output tokens cost 3-5x more than input tokens. Optimizing output length (through better prompting, constrained decoding, or distillation) has outsized cost impact.

3. The build-vs-buy crossover is around 3-5M tokens per day. Below this, APIs win on total cost of ownership. Above this, self-hosting wins — and the margin widens rapidly with volume.

4. Distillation is the most underappreciated lever. A task-specific 7B model at USD 0.05/M tokens can replace a general 70B model at USD 0.46/M tokens for the vast majority of production workloads.

5. Costs are falling 10x every 2 years. Hardware improvements (HBM bandwidth, FP4 tensor cores) compound with algorithmic improvements (MoE, better attention, KV cache compression) to drive relentless cost reduction.

The cost of LLM inference is dropping rapidly, but the demands on inference are growing even faster. Reasoning models that generate 50,000 tokens per query, agent loops that make dozens of LLM calls per task, and always-on processing pipelines that consume tokens continuously — all of these push total spending upward even as unit costs plummet. The winners will be those who understand the math well enough to operate on the efficient frontier — squeezing maximum quality from every dollar of compute.