New: Run AI anywhere your cloud runs
Learn more
H100 cloud rental prices have dropped from 2023 peaks - compare 41 providers and cost-per-token benchmarks
Three years after launch, the H100 is no longer scarce, no longer cutting-edge, and no longer expensive — unless you're renting one from a hyperscaler. Here's how the market actually works.
In the spring of 2023, an NVIDIA H100 renting an H100 for a month of continuous use cost roughly as much as a used car payment. AWS's P5 instances listed at over $60 per hour for an 8-GPU node — roughly $7.50 per GPU-hour — and Google Cloud's A3 instances cleared $11 per GPU-hour. Hyperscalers and AI infrastructure providers pre-committed large portions of global H100 supply, effectively locking down entire GPU batches and creating multi-month allocation queues.
That market is gone. As of May 2026, the cheapest publicly listed H100 spot instance sits at $1.25 per hour. The most expensive on-demand H100 — on Microsoft Azure — runs $12.29 per hour.
A twelve fold price variance for the exact same chip — the only variable is who's selling it. This guide cuts through the twelve fold variance. Continue reading to learn how to find the right H100 at the right price.
Three forces collapsed H100 pricing. First, supply caught up: TSMC expanded CoWoS packaging capacity through 2024, and refurbished cards started flowing into secondary markets.
Second, competition arrived from below — Vast.ai, RunPod, Hyperbolic, and dozens of specialized GPU clouds built lean, GPU-only businesses that hyperscalers could not match on raw price.
Third, the next generation arrived. With B200 (Blackwell) shipping from late-2025 and B300 from January 2026, the H100 is no longer the frontier chip, and frontier-chip pricing power has migrated up the stack.
AWS's June 2025 decision to cut P5 prices by 44% was the catalytic moment. Within weeks, the entire industry repriced. Today's pricing landscape looks nothing like the launch market.
The honest summary: If you want the cheapest H100, you can find one on a specialized provider for $1.49–$2.10/hr. If you want enterprise SLAs and integrated tooling, expect to pay $3.50–$7/hr on hyperscalers. Both options serve real markets. Both are reasonable for different teams.
Here is the current landscape, normalized to single-GPU equivalent pricing on H100 80 GB instances. Where providers only list 8-GPU nodes, the per-GPU rate is shown.
Across 40+ tracked providers, the median on-demand rate sits at roughly $3.61 per GPU-hour, but that average obscures a bimodal distribution. The specialist clouds cluster around $1.50–$2.50/hr; hyperscalers cluster around $4–$7/hr. Very little sits in between.
A practical complication that pricing tables don't capture: most production teams don't get to pick one provider. Compliance teams require AWS or Azure for certain workloads.
Data residency rules in Europe push other workloads onto EU-native providers. Cost-sensitive batch jobs migrate to whatever is cheapest that week.
The result is that the same organization often runs H100 instances across three or four clouds simultaneously — each with its own provisioning console, billing format, IAM model, and driver-image workflow. The price spread is real, but so is the operational cost of capturing it. Multi-cloud control planes such as emma's GPU virtual machines have emerged specifically to make that arbitrage practical: a single provisioning flow across supported cloud environments, with pre-validated CUDA images so each new VM doesn't begin with a half-day of driver debugging.
Whether you adopt a platform layer or build the orchestration in-house, the operational complexity of multi-cloud GPU fleets is now a real line item — sometimes larger than the hourly rate savings that motivated the multi-cloud setup in the first place.
The hourly rate of a GPU tells you very little about whether it is “cheap” for a given workload. What matters in practice is cost per million tokens (inference) or cost per training step (training). Both are dominated not by sticker price, but by how efficiently the hardware is utilized.
In real production systems, utilization can vary widely—often from below 10% in poorly optimized deployments to 60–90% in well-batched serving stacks—and that gap drives order-of-magnitude differences in effective cost.
"Paying for an H100 running at 10% load transforms $0.013 per thousand tokens into $0.13 — more expensive than premium APIs." — Introl, Inference Unit Economics, 2026
Consider a 70B-parameter model served on an H100 SXM. In FP16, the model does not fit on a single GPU (it requires tensor parallelism across multiple GPUs due to memory limits). In practice, such a model is typically served using 2–8 GPUs depending on precision, context length, and throughput requirements. At low batch sizes (e.g., batch size 1) and naive decoding without optimized kernels, GPU utilization remains low (<10%), and system-level cost per million tokens can exceed tens of dollars due to poor throughput efficiency.
When the same deployment is optimized using continuous batching, FP8 or INT8 quantization, FlashAttention, and speculative decoding, effective throughput can improve by approximately 3–8× depending on workload and request distribution. This increases GPU utilization and amortizes fixed compute overhead across more tokens, reducing cost per million tokens proportionally. In well-optimized, high-utilization serving environments, a 70B model on H100-class infrastructure can reach roughly ~$5–$15 per million tokens, with lower values achievable only under sustained high throughput and ideal batching conditions.
The key driver is not raw hardware cost, but utilization efficiency: inference economics are dominated by how effectively compute is saturated, not the sticker price of the GPU.
That’s why H100 vs A100 comparison does not have a single universal winner because cost-per-token depends heavily on utilization and serving efficiency. At very low utilization and poor batching efficiency, a lower-cost A100 instance can sometimes appear more cost-effective due to its lower hourly price. However, as workload efficiency improves, the higher memory bandwidth and optimized inference stack of the H100 generally result in significantly better throughput per dollar for 70B-class models. The real determinant is not the GPU price, but how effectively the system saturates memory bandwidth and batching capacity under load.
Knowing your real utilization matters more than the sticker price. Yet most teams have no centralized view of GPU utilization across clouds — utilization data lives in CloudWatch on AWS, in Cloud Monitoring on GCP, in Azure Monitor, and in nothing at all on most specialist providers.
Without GPU metrics (utilization, vRAM usage, vRAM activity) visible alongside CPU and memory in a single monitoring pane, the "$2.99/hr is more expensive than $1.49/hr" reasoning continues unchallenged. The 100× difference between configured and unconfigured H100s never surfaces.
Most pricing comparisons treat on-demand as the default. For cost-conscious teams, this is almost always wrong. The real cost-optimization choice is between three modes:
Roughly 40–60% cheaper than on-demand. Available when provider capacity is underutilized. The catch: instances can be reclaimed with 30–120 seconds of warning. Ideal for batch inference, fine-tuning with frequent checkpointing, hyperparameter sweeps, and experimentation.
Cost-sensitive AI labs run essentially everything on spot.
Pay-as-you-go, no commitment. Right choice for production inference where preemption is unacceptable, demos, irregular workloads, and any time you genuinely cannot tolerate interruption.
Commit to 1–3 years; cloud providers respond with steep discounts. Lambda Labs' reserved H100 rate of $1.89/hr is approaching marketplace spot pricing.
CoreWeave's 60% reserved discount on $6.16/hr brings the effective rate near $2.50/hr with enterprise SLAs and InfiniBand networking thrown in.
Procurement heuristic. If you need GPUs less than 40 hours per month, stay on-demand on a specialist. If you need 200+ hours per month consistently, run reserved on a mid-tier provider.
If you need 600+ hours per month and you have engineering capacity to manage preemption, run spot.
The Hopper-Blackwell transition makes 2026 a peculiar moment for GPU procurement. Four meaningful options exist:
GPU selection is not determined by model size thresholds, but by system bottlenecks: compute throughput, memory bandwidth, interconnect efficiency, and achievable utilization under real traffic patterns.
A100-class GPUs remain cost-effective for smaller and quantized models, while H100 dominates general-purpose training and inference scaling. H200 improves memory-bound inference stability, and Blackwell-class GPUs (B200) extend performance primarily in large-scale, bandwidth- and interconnect-constrained workloads such as frontier model training and high-throughput serving.
Data egress fees on hyperscalers run $0.08–$0.12 per GB. Moving a 2 TB training dataset out of AWS costs more than $160. For multi-cloud or hybrid workflows, egress can dwarf compute.
Private cross-cloud networking backbones — the kind that connect a training cluster on one provider to inference on another without routing through the public internet — can cut these costs by 60–80%, though they require explicit setup.
Startup latency on some providers exceeds 10 minutes per instance — fatal for autoscaling inference. Specialist clouds typically launch in 30–90 seconds; hyperscalers can take 5–15 minutes for large GPU nodes. Even worse: launches that succeed but boot with broken drivers. Driver mismatch remains the single most common cause of failed GPU jobs, which is why pre-validated CUDA images (NVIDIA driver + CUDA toolkit + framework dependencies, baked into the OS image) are increasingly table stakes for production teams.
Networking quality varies enormously. InfiniBand-class interconnects (CoreWeave, hyperscalers) deliver near-linear multi-node scaling. Ethernet-only marketplace providers can lose 30–50% of multi-node throughput to communication overhead.
Storage costs on managed platforms are often bundled but capped. Bringing your own dataset to a $1.50/hr marketplace instance may require attaching block storage at $0.10–$0.20 per GB per month.
Governance gaps. GPU VMs procured ad-hoc by data science teams frequently fall outside the same RBAC, tagging, and cost-attribution policies that govern the rest of an organization's infrastructure.
The result is shadow GPU spend — instances running without a tagged owner, instances provisioned outside approved providers, and a CFO who finds out about $40,000 of unattributed compute at quarter-end. For regulated workloads in the EU, this is no longer just a budget problem: NIS2 and DORA both treat untracked compute as an audit finding.
An H100 SXM5 purchased outright for $30,000 reaches break-even against $2.00/hr cloud rental after roughly 15,000 GPU-hours of utilization — about 21 months of continuous use, or 4 years at 50% utilization.
That's before infrastructure (power, cooling, networking: typically $5,000–$50,000), operations staff, and depreciation. By the time you've amortized an H100, the B200 is mid-cycle and the Rubin generation is approaching.
Reality check. Most organizations running fewer than 24/7 workloads on a single GPU for 18+ months are better served by cloud. Buy only if you have predictable, high-utilization workloads, in-house infrastructure expertise, and a clear depreciation strategy. The 2023 logic of "buy because you can't get cloud capacity" no longer applies.
First, Blackwell volume continues to ramp. As B200 and B300 capacity comes online, frontier workloads will migrate up the stack, leaving more H100 inventory available for inference and mid-scale training. Expect continued downward pressure on H100 spot pricing, possibly into the $1.00–$1.50/hr range by late 2026.
Second, HBM memory pricing has been rising — Samsung and SK Hynix raised HBM3e prices roughly 20% for 2026 contracts. This pressures Blackwell margins more than Hopper, since the H100 uses older HBM3, which has more mature supply.
Third, NVIDIA's Rubin architecture is expected in late 2026, which will further commoditize earlier generations. The H100 will likely remain the cost-optimal choice for most production workloads through mid-2027, after which Blackwell pricing should normalize. We track GPU roadmap and pricing movements in our quarterly cloud GPU market reports.
For most production AI in 2026, the H100 is no longer the bleeding edge — it's the obvious choice.
H100 cloud rental prices range from $1.25 per hour (spot pricing on Vast.ai and similar marketplaces) to $12.29 per hour on Microsoft Azure. The market average across 40+ tracked providers is around $3.61 per GPU-hour.
Specialized GPU clouds typically charge $1.49–$2.69/hr on-demand, while AWS, Azure, and Google Cloud charge $3–$7/hr or more for equivalent hardware. See the full pricing comparison table above.
Specialized providers like Vast.ai, Hyperbolic, RunPod, and Lambda Labs run lean, GPU-only operations. They skip the bundled ecosystem services — managed ML platforms, enterprise networking, SOC compliance, integrated MLOps tooling — that hyperscalers include.
For raw GPU access, this makes them 3–10× cheaper. Hyperscalers justify the premium with InfiniBand-class networking, integrated tooling, enterprise SLAs, and procurement contracts that large organizations require.
Spot H100 instances run 40–60% cheaper than on-demand — typically $0.73–$1.50/hr versus $2.00–$3.50/hr on-demand. Spot is ideal for interruptible workloads: batch inference, fine-tuning with checkpointing, hyperparameter sweeps, and experimentation.
Avoid spot for production inference, time-sensitive training, or any job that cannot recover from preemption. See our procurement heuristic for choosing between modes.
Is the H100 still worth renting in 2026 with B200 available?
For most workloads, yes. The H100 has become the cost-performance sweet spot as supply caught up and prices dropped 75% from 2023 peaks. The B200 offers higher throughput but commands a 2x price premium with less mature deployment tooling.
For training models under 70B parameters and standard inference workloads, the H100 typically delivers lower cost-per-token than newer Blackwell GPUs. Expect this to hold through at least mid-2027.
Training a 70B-parameter model from scratch typically requires on the order of 300,000 to 1.5 million H100 GPU-hours, depending on token budget, training efficiency, and architecture optimization.
At approximately $2–$3 per GPU-hour on specialized providers, this corresponds to ~$0.7M to $3.5M in compute cost alone, excluding storage, networking, failed runs, and evaluation overhead.
Fine-tuning the same model with LoRA or QLoRA is significantly cheaper. A typical configuration using ~8 H100 GPUs for 300–1,000 hours results in ~$5,000 to $20,000 in compute cost.
H100 PCIe vs SXM5: which should I rent?
SXM and PCIe H100s differ mainly in interconnect and scaling efficiency, not raw single-GPU performance. NVIDIA H100 Tensor Core GPU SXM variants use NVLink/NVSwitch with ~900 GB/s-class GPU-to-GPU bandwidth, while PCIe 5.0 is limited to ~64 GB/s, making SXM significantly better for multi-GPU training and tensor parallelism.
PCIe instances are typically 10–25% cheaper and suitable for single-GPU inference or low-communication workloads. For 4–8 GPU training runs, SXM is strongly preferred because communication overhead becomes a bottleneck. The practical rule: buy PCIe for cost-efficient inference, rent SXM for scalable training and tightly coupled distributed workloads where bandwidth dominates performance.
Teams running H100 workloads across more than one cloud — common for cost optimization, data residency, or capacity reasons — typically face fragmented tooling: separate consoles per provider, inconsistent driver and CUDA images, isolated cost and monitoring views, and varied governance models.
Options include building custom Terraform modules and tagging conventions internally, or using a multi-cloud control plane such as emma's GPU virtual machines, which consolidates provisioning, governance, and cost attribution across AWS, GCP, Azure, and EU-native providers behind a single workflow.
The right answer depends on team size and how much operational lift you want to absorb internally.
A single H100 SXM5 or PCIe card costs $25,000–$40,000 as of 2026, down from $40,000+ peaks on secondary markets in late 2023.
A complete 8-GPU HGX H100 system with networking and cooling runs $300,000–$500,000. Add infrastructure ($5,000–$50,000), power ($60/month per GPU), and ongoing operations costs when compared to cloud.
Pricing verified across primary provider pricing pages and independent trackers as of May 2026.
