GPU & hardware sizing cheat sheet
A working formula for VRAM sizing, a tiered GPU table you can use as a first-pass proposal reference, and cheaper alternatives for narrow, single-task agents.
Curated by Kailas Lovlekar · July 2026. AI models, hardware and pricing are evolving at a rapid pace — information in this article is for broad reference only.
This is the question infrastructure teams ask once model choice (Part 1) and governance (Part 2) are settled: what do we actually need to buy? The figures below are a planning-stage reference, not a vendor quote — always validate against a proof-of-concept load test before committing capital.
The sizing formula
Total VRAM required breaks into three components:
Total VRAM = Model Weights + KV Cache (concurrency & context) + System/CUDA overhead
Model weights, by precision
- FP16 / BF16 (full precision) — roughly 2GB per billion parameters
- FP8 / INT8 (standard enterprise quantization) — roughly 1GB per billion parameters
- INT4 / AWQ (high-density quantization) — roughly 0.5GB per billion parameters
KV cache & context overhead
Model weights are fixed, but the KV cache grows with concurrent users and context length. As a rule of thumb, reserve an additional 20–50% of VRAM on top of weights for KV cache and CUDA buffers, scaling toward the higher end for longer context windows (32k+) or higher concurrent batch sizes.
Tiered GPU reference table
| Model tier | Precision | Weights VRAM | Buffer + KV cache | Recommended GPU setup | Typical banking use |
|---|---|---|---|---|---|
| Small (3B–3.8B) | FP16 | ~8GB | ~8–12GB | 1x L40S 48GB (or 1x A100 40GB) | Fast classification, low-latency micro-services, desktop co-pilots |
| Mid-size (7B–8B) | FP16 / FP8 | ~16GB / ~8GB | ~16–20GB | 1x L40S 48GB | Internal RAG search, document summarisation, SQL/code completion |
| Large (14B–32B) | FP8 / INT8 | ~32GB | ~16–32GB | 1x A100/H100 80GB (or 2x L40S tensor-parallel) | Financial report analysis, structured data extraction, multi-turn chat |
| Enterprise standard (70B–72B) | FP8 / INT8 | ~72–80GB | ~32–48GB | 2x H100 80GB (or 2x A100 80GB) | Core enterprise LLM, quantitative reasoning, bank-wide RAG backend |
| Enterprise uncompromised (70B–72B) | FP16 | ~140–145GB | ~40–60GB | 2x H200 141GB (or 4x H100 80GB) | High-precision audit, risk modelling, heavy concurrent traffic |
| Deep reasoning MoE (e.g. 671B-class) | FP8 / INT4 | ~350–400GB | ~80–120GB | 8x H100/H200 (full HGX node) | Quantitative strategy simulation, multi-step risk auditing |
Indicative GPU pricing
Enterprise GPU pricing moves with supply and generation cycles — treat this as directional, not a quote, and confirm current street pricing with a hardware vendor at proposal time.
| GPU | VRAM | Form factor | Indicative price (mid-2026) |
|---|---|---|---|
| NVIDIA H200 SXM | 141GB | Server board (HGX) | ~$35,000–40,000 |
| NVIDIA H100 PCIe/SXM | 80GB | Server card | ~$22,000–30,000 |
| NVIDIA L40S | 48GB | Server card (PCIe) | ~$7,500–9,500 |
| NVIDIA RTX 6000 Ada | 48GB | Workstation (PCIe) | ~$6,800–7,500 |
| NVIDIA RTX 4090 (consumer) | 24GB | PCIe card | ~$1,800–2,200 |
| Apple Mac Studio (M-series Ultra) | 128–192GB unified | Desktop workstation | ~$4,500–7,000 |
The RTX 4090 and Mac Studio are lab/proof-of-concept hardware — no ECC memory or datacenter support, not for production banking workloads.
GPU & hardware availability in the GCC
Since most PhantomOps deployments sit inside the GCC, where the compute actually comes from matters as much as which GPU you pick on paper. Availability splits sharply by scale, and it's moved fast through 2026.
Large-scale (H100 / H200 / GB300-class) — through sovereign cloud or a licensed local partner
Direct import of hyperscale AI chips into the Gulf sits inside US export-control policy, which has shifted materially more than once in the past year:
- UAE — elevated to US Commerce Department Country Group A:5 in July 2026, giving approved entities licence-free access to advanced Nvidia chips, including Blackwell-generation parts. Core42 (G42) already runs H100 in-country and is deploying H200; e& UAE and Core42 launched a sovereign GPU compute platform in July 2026; Microsoft's Azure partnership with G42 adds further H100/H200 capacity via Khazna data centres.
- Saudi Arabia — not on the licence-free list, but specific large deals are cleared case by case. Humain (under the Public Investment Fund) has a Nvidia partnership for up to 600,000 GPUs over three years, with an initial 18,000-chip GB300 (Blackwell) tranche going into Riyadh and Dammam sites from Q2 2026.
- Qatar — MEEZA and a Google Cloud region anchor a sovereign banking/government cloud hub in Doha.
- Kuwait — Ooredoo Kuwait's first sovereign AI data centre already runs on Nvidia H200; Google Cloud and Microsoft Azure have both announced upcoming Kuwait cloud regions.
- Bahrain — stc/center3's Bahrain Data Center Park is under construction as part of a wider regional connectivity investment.
- Oman — data centre capacity is growing steadily under the national AI and Digital Future Program, though no major H100/H200-class deployment has been announced there yet.
For any workload needing 70B+ models or H100/H200-class hardware, the practical route for a bank today is a regional sovereign cloud or licensed local partner — Core42/e& in the UAE, Humain in Saudi Arabia — rather than direct chip import. This is true for export-control reasons and because these providers already offer the in-country data-residency guarantees a BFSI compliance team will ask for.
Card-level (L40S / RTX 6000 Ada, 48GB-class) — available off the shelf
For the 48GB-class cards this guide recommends for most narrow, single-task agents, regional IT distributors already stock and ship them commercially — authorised Nvidia partners and resellers in Dubai and across the UAE and Saudi Arabia list L40S, RTX 6000 Ada and comparable cards with standard delivery and manufacturer warranty. This tier doesn't need the sovereign-cloud route; a normal IT procurement cycle through an authorised distributor is enough.
Export-control classification and licensing requirements for advanced AI chips in the Gulf have changed more than once in 2025–2026 and are likely to keep moving. Confirm current status with your compliance and procurement team before finalising a specific chip, quantity and country in a hardware proposal.
Don't over-provision: cheaper paths for SLMs and single-task agents
A dedicated document-extraction agent or a KYC field-checker doesn't need an H100. Most of PhantomOps' narrow, single-task agents run comfortably on far cheaper hardware:
Multi-agent on one 48GB card
A single L40S or RTX 6000 Ada (~$7,000–9,500 CapEx) can host three or four small models simultaneously via a multi-model serving engine like vLLM — for example, a 3B SQL-parsing agent (~8GB), a 3.8B risk-classification agent (~8GB), and an 8B document-summarising agent (~10GB), with the remaining ~20GB reserved for concurrent KV cache.
Single-task workstation node
For an isolated, department-level deployment (compliance, back-office), one RTX 6000 Ada or two RTX 4000 Ada cards (~$3,000–7,200 total) comfortably runs a 3B-class model at well over 100 tokens/second — effectively instantaneous for a single, narrow task.
CPU-only for batch, non-real-time work
For overnight or batch agentic tasks with no human waiting on a live response (auditing a trade log, batch-classifying documents), a modern server CPU with high-bandwidth RAM (128GB+) running a quantized model via llama.cpp eliminates GPU CapEx entirely — at the cost of throughput, which is an acceptable trade-off for background work.
Reserve H100/H200-class hardware for genuinely large models (70B+) or deep-reasoning MoE workloads. For the majority of narrow, single-task banking agents, a 48GB-class card is both sufficient and roughly a quarter of the capital cost.
Two design principles worth including in any hardware proposal
- Plan for redundancy. Production banking services shouldn't run on a single card. If you're proposing 2x H100 for a 70B model, budget two identical nodes behind a load balancer for failover (N+1 at minimum).
- Use NVLink/SXM, not PCIe, above 32B. Multi-GPU tensor parallelism for models 32B and larger needs high-bandwidth GPU-to-GPU interconnect to avoid latency bottlenecks — PCIe alone becomes the constraint at that scale.
Where PhantomOps lands
We size every deployment against the actual workload, not the largest model available. Most BFSI pilots start on a single 48GB-class card, sourced directly from a regional distributor, running one or two agents; enterprise-wide rollouts move to a dual-H100 or H200 backend — provisioned through a sovereign cloud or licensed local partner rather than direct import — only once usage data justifies it, validated through a proof-of-concept load test before any hardware purchase is finalised.
Sources: NVIDIA GPU specifications and enterprise pricing surveys (2026); vLLM and NVIDIA Triton documentation on quantization and tensor parallelism. GCC availability: US Commerce Department Country Group A:5 designation for the UAE (July 2026, reported by Arabian Business and U.S. News); Core42/e& UAE sovereign GPU platform announcements; NVIDIA Newsroom on the Humain (Saudi PIF) partnership; Ooredoo Kuwait sovereign AI data centre announcement; regional Nvidia distributor listings (UAE/Saudi Arabia). Pricing and availability figures are indicative as of mid-2026 — confirm current status with a hardware vendor and your compliance team before budgeting.