Self-Hosting LLMs: Hardware and Economics

Self-hosting an LLM is not a hardware decision first. It is a utilization bet. The same GPU running the same model can cost you 25 to 75 times more or less per million tokens depending entirely on how busy it is — and a GPU that sits mostly idle, which describes most individual and small-team self-hosting setups, almost never beats a flat monthly subscription on raw token economics. This page works through the actual numbers: how much memory a model needs at each precision, why context length blows that budget up further than people expect, what the hardware options cost at each rung of the ladder, and what a million self-hosted tokens really costs compared to the consumer AI subscriptions this wiki already prices out.

The hardware ladder

Four rungs cover almost every self-hosting decision, from a single consumer card to renting whatever a cloud provider has in stock. Capability, cost, and commitment all rise together.

Tier 1: consumer GPU

The RTX 4090 (24 GB GDDR6X) and RTX 5090 (32 GB GDDR7, 1.79 TB/s bandwidth) are the current top of NVIDIA’s gaming line and the default entry point for local LLM work. The RTX 5090 launched at a $1,999 MSRP in January 2025; NVIDIA production of the RTX 4090 ended in late 2024, and both cards have traded well above their original list prices through 2026 on constrained supply and AI demand, with the RTX 5090 frequently listed above $2,500 and sometimes above $4,500 at retail.¹² A single card in this tier comfortably runs 7B–13B models at INT4 or INT8, and 34B at INT4 with careful memory management.

Tier 2: prosumer / workstation

The RTX 6000 Ada Generation (48 GB GDDR6, $7,350 at launch) and its Ampere predecessor the RTX A6000 (48 GB, roughly $4,650 new) are workstation cards built for exactly this: enough VRAM in one slot to hold a 34B model at FP16 or a 70B model at INT4 with room for a KV cache.³⁴

Apple’s Mac Studio is a fundamentally different machine, not a discrete GPU, and it belongs on this rung for a different reason: unified memory. The CPU, GPU, and Neural Engine share one memory pool, so a model’s weights sit in a single address space instead of being split across VRAM and system RAM. As of the M5 Ultra refresh announced by Apple on August 25, 2026, that pool scales up to 512 GB, with the base M5 Ultra Mac Studio starting at $5,499 and a currently orderable maximum configuration (256 GB memory, 16 TB storage) at $18,299; the 512 GB configuration was announced but not yet priced at the time of writing, with Apple stating only that it ships “in late October” 2026.⁵ The tradeoff is bandwidth: M5 Ultra’s unified memory runs at 1.2 TB/s, against 3.35 TB/s on an H100 and roughly 2 TB/s on an A100.⁶⁷⁸ A Mac Studio can hold a model no single GPU in this tier can fit — but token generation is memory-bandwidth-bound, so it will generate those tokens more slowly than a card with a fraction of the capacity and several times the bandwidth. It is the right answer when the question is “can this fit at all,” not “how fast can this serve concurrent users.”

Tier 3: single-node on-prem server

An 8-GPU server pools VRAM across cards over NVLink or a similar fast interconnect, giving 640 GB of usable VRAM in an 80 GB×8 configuration — enough for the largest open-weight models at full precision. NVIDIA does not publish list prices for these systems; market estimates place a full 8× A100 80GB DGX-class system at roughly $150,000–$200,000, and an 8× H100 80GB system (DGX-class or third-party HGX build) at roughly $250,000–$450,000, varying with networking, storage, and support contract.⁹¹⁰ Add facilities: GPU servers in this class draw 2–5 kW and annual power-and-cooling can run 20–30% of the hardware cost.¹¹ This tier is a multi-year capital commitment, not a purchase you make to test an idea.

Tier 4: rented cloud GPU

Renting sidesteps the capital outlay and the 3–4 year refresh cycle entirely — you pay only for the hours you use, on hardware someone else maintains. RunPod and Vast.ai run marketplace-style pricing that gets an RTX 4090 down to roughly $0.29–$0.74/GPU-hour and an H100 SXM to roughly $2.69–$3.29/GPU-hour depending on reliability tier.¹²¹³ Lambda prices on-demand H100 SXM at $3.99–$4.29/GPU-hour depending on cluster size (cheaper per-GPU in an 8-GPU instance than a single-GPU one), with A100 80GB — sold only as an 8-GPU instance — at $2.79/GPU-hour.¹⁴ Nebius , CoreWeave , Crusoe , and Paperspace round out the field with their own mixes of managed platform, top-tier reliability, energy-optimized fleets, and notebook-first tooling respectively — see GPU clouds and neoclouds compared for how they differ. This tier is where the utilization argument later on this page matters most, because rental cost scales with hours used automatically — there is no idle fixed cost to amortize, which is precisely the opposite failure mode from owned hardware.

Sizing VRAM: the formula, worked

Weights. VRAM for weights follows one formula: parameters (billions) × bytes-per-parameter. FP16/BF16 uses 2 bytes/parameter, INT8 uses 1, INT4 uses 0.5.

VRAM (GB) = params (billions) × bytes-per-parameter

That is weights only. Add 15–40% on top for KV cache, activations, and runtime/framework overhead — presenting the weight-only number as “what you need” is the single most common sizing mistake. Applying that range to the table above:

Read straight off this table against the hardware ladder above: a 24–32 GB consumer card fits 7B at any precision and 13B at INT8/INT4. A 48 GB workstation card comfortably fits 34B at INT8 (39.1–47.6 GB) across the whole overhead range, but 70B at INT4 (40.3–49 GB) only fits at the low end of that range — meaning a short context window. Push the context out and 70B INT4 spills past 48 GB, which is exactly the KV-cache effect covered next.

The KV cache: why context length changes the math

The formula above sizes the model. It says nothing about the conversation. Every token generated keeps a key and value vector cached per layer, per attention head, so a long conversation or a large retrieved-document context adds memory on top of the weights, independent of model precision:

KV cache (bytes) = 2 × L × H_kv × d × T × bytes-per-element
  L      = number of transformer layers
  H_kv   = number of key/value heads
  d      = head dimension
  T      = sequence length (tokens)
  2      = one Key tensor + one Value tensor per token

The detail that trips people up is H_kv. Under plain multi-head attention it equals the model’s total attention head count. But most current open-weight models use grouped-query attention (GQA), where several query heads share one key/value head specifically to shrink this cache — H_kv is then far smaller than the head count you’d read off the model card at a glance.¹⁵ Llama-3-70B is a working example: 80 layers, 64 query heads, but only 8 key/value heads, head dimension 128, published in its Hugging Face config.¹⁶

Worked at BF16 (2 bytes/element):

Per-token KV cache = 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 0.31 MB/token

At 4,096-token context:   327,680 × 4,096   ≈ 1.25 GB   (one sequence)
At 128,000-token context: 327,680 × 128,000 ≈ 39.1 GB   (one sequence)

At a short context, the KV cache is a rounding error next to the weights. At a long context, it is not: one 128K-token sequence on Llama-3-70B needs about 39 GB of cache — more memory than the entire model’s weights at INT4 (35 GB, from the table above), and that is per concurrent sequence, so a handful of simultaneous long-context requests can exceed the weight footprint several times over. And that 39 GB figure is the reduced number: had this model used plain multi-head attention instead of GQA (H_kv = 64 instead of 8), the same sequence would need roughly 313 GB, which is why GQA is now close to universal in production model architectures rather than an optional optimization. This is the concrete, verifiable version of “long-context serving needs meaningfully more memory than the weights alone” — run the arithmetic yourself against a model’s own config.json before sizing hardware for a long-context deployment. See GPU scheduling for AI for how serving engines like vLLM manage this cache across concurrent requests via paged attention.

The precision tradeoff is real, not free

INT8 quantization is close to a free lunch for most applications: a peer-reviewed evaluation of well-tuned INT8 weight-and-activation quantization (W8A8) found accuracy degradation of only 1–3% against the full-precision baseline across a broad benchmark suite.¹⁷ INT4 is a different story once the task involves multi-step reasoning. The same body of work found INT4 weight-only quantization “more competitive than expected” on general benchmarks — but a separate study focused specifically on mathematical reasoning (GSM8K, MATH, AIME) found post-training INT4 quantization causing accuracy drops of up to 69.81% in its harder evaluation settings, concentrated in procedural/execution errors rather than conceptual mistakes.¹⁸ Treat INT4 as safe for retrieval, summarization, and classification-style workloads, and validate it explicitly — against your own evaluation set, not a general leaderboard — before shipping it on anything reasoning-heavy. See Quantization and Hardware constraints for AI systems for the underlying mechanics.

The self-hosted cost model

Two cases need different arithmetic: hardware you own, where cost is fixed regardless of use, and hardware you rent, where cost scales with hours consumed automatically.

Owned hardware. The purchase price is sunk over the depreciation period whether or not the card is running. Spread it over calendar hours in that period, then divide by the fraction of hours actually spent generating tokens (utilization, U) to get the fixed-cost component per hour of actual use. Power is metered only while running.

$/hour-of-use = [ price / (years × 8,760 hours × U) ]  +  (TDP in kW × $/kWh)

Rented hardware. No amortization term — the provider already priced depreciation, power, and their margin into the hourly rate:

$/hour-of-use = rental $/GPU-hour

Either way, convert to a cost per million tokens with the model’s sustained generation throughput:

$ per million tokens = ($/hour-of-use ÷ (tokens/second × 3,600)) × 1,000,000

Worked example 1: an individual, owned GPU

Take an RTX 5090 bought at its $1,999 MSRP, depreciated over 3 years (26,280 hours) — the refresh cycle this wiki’s on-premise comparison also uses for GPU hardware.¹¹ TDP is 575 W (0.575 kW).¹ Electricity at the June 2026 US average residential rate of 18.34¢/kWh gives a power cost of 0.575 kW × $0.1834/kWh ≈ $0.105/hour while running.¹⁹ For throughput, published community benchmarks for an 8B-class model at 4-bit quantization on this GPU class cluster loosely around 100–150 single-stream tokens/second, varying widely by inference engine, context length, and exact quantization method; this example uses a conservative 100 tokens/second — plug in your own measured number to refine it.²⁰

The fixed hardware cost does not change — the card cost $1,999 whether it ran once or continuously. What changes is how many tokens it generated during the period over which that cost is spread, so cost per token rises sharply as utilization falls. This is the same “does it run continuously or intermittently” logic this wiki uses for automation and on-premise-vs-cloud decisions generally; see When automation pays for itself and On-premise vs cloud for AI workloads for the general argument. What this table adds is the specific self-hosted-LLM numbers.

Worked example 2: rented GPU at scale — utilization decides everything

Rent a single H100 SXM at a representative $3/GPU-hour (within the $2.69–$4.29 range quoted above) to serve a 70B-class model. Throughput here depends almost entirely on concurrency:

Low concurrency (interactive, one request at a time). Published single-stream benchmarks for a 70B model at INT4 (AWQ) on an A100/H100-class GPU cluster around 35–45 tokens/second.²¹ At 40 tokens/second:

tokens/hour = 40 × 3,600 = 144,000
$/million tokens = ($3 ÷ 144,000) × 1,000,000 = $20.83

High concurrency, continuous batching (many simultaneous requests, well-tuned serving). A serving engine that batches many concurrent requests together — this is what vLLM and similar engines exist to do — pushes far more tokens through the same GPU per hour. Batched, offline-style throughput benchmarks for a 70B model on a single H100 have been reported in the 1,000–3,000+ tokens/second range depending on precision and batch composition, including a widely reported MLPerf Inference v4.1 submission.²² Taking that range:

At 1,000 tok/s:  ($3 ÷ 3,600,000) × 1,000,000 = $0.83/million tokens
At 3,000 tok/s:  ($3 ÷ 10,800,000) × 1,000,000 = $0.28/million tokens

Same GPU, same rental rate, same model class — a 25 to 75x swing in cost per token, driven entirely by whether the serving stack keeps the GPU saturated with concurrent work. This is why GPU scheduling and serving-engine choice (vLLM vs. simpler runtimes like Ollama ) are cost decisions, not just performance decisions — see Reducing LLM inference costs in production for the operational tactics that get you toward the batched end of that range.

Self-hosted vs. consumer subscription: the actual numbers

This wiki’s own pricing page puts Claude Pro at $20/month (monthly billing) or $17/month (annual), ChatGPT Plus at €23/month, and Google AI Pro at €21.99/month, as of June 2026 — flat fees with capped, unmetered usage rather than per-token billing.²³ Anthropic’s current API rate for its mid-tier model (Claude Sonnet 5) is $2 input / $10 output per million tokens.²⁴

Take the owned-GPU numbers from worked example 1 and ask: at $20/month, how many self-hosted tokens does that buy at compute-only cost?

A human typing into a chat interface — even for hours a day — is not generating tens of millions of tokens a month; the workload this wiki’s own subscription-pricing page uses as its “heavy automated” example totals about 25 million tokens across 10,000 API calls in a month.²³ An individual’s GPU spends most of its calendar hours idle between prompts, which lands realistically in the 1–5% utilization band above, not 100%. The honest conclusion: for an individual chatting with a model, self-hosting essentially never beats a $20/month subscription on pure token economics, unless the hardware is already paid for and busy with something else (gaming, local image generation, other workloads), making the marginal cost of also serving your own chat close to zero. The reasons to self-host at this scale are real — data never leaves your machine, no rate limits, offline capability, full control over the exact model and its behavior — but none of them are “it’s cheaper per token.” See AI subscription and pricing models in 2026 for the full consumer-tier breakdown and Managed vs. reserved vs. self-hosted inference for how this changes once you’re serving other people, not just yourself.

The picture inverts for worked example 2’s high-concurrency case: $0.28–$0.83 per million tokens undercuts Sonnet 5’s $2–$10 API pricing by roughly 2x to 36x — but only at sustained, well-batched, high-concurrency utilization, which describes an operator serving many users through automated pipelines, not an individual’s chat session. That is the same “continuous vs. intermittent” divide this wiki’s automation-economics reasoning applies elsewhere, applied here specifically to self-hosted token generation: see When automation pays for itself for the general version of that argument and LLM cost optimization for cutting the cost further on either side of it.

What this math doesn’t capture

Every number above is compute cost only. It excludes the engineer-hours to stand up and maintain a serving stack, on-call burden when a GPU node fails at 2am, security patching, redundancy for uptime, and the capability gap between the open-weight model you can run and the frontier proprietary model an API gives you access to for the same token price. A self-hosting decision that only pencils out on the compute-cost line and ignores these is incomplete — see AI total cost of ownership for folding operational cost into the comparison properly.

Sources

1. NVIDIA, “GeForce RTX 5090” product specifications: https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/ — 32 GB GDDR7, 575W TDP, $1,999 MSRP, January 2025 launch.
2. VideoCardz, “GeForce RTX 5090 turns one, now costs nearly twice the MSRP”: https://videocardz.com/newz/geforce-rtx-5090-turns-one-now-costs-nearly-twice-the-msrp — 2026 street pricing.
3. VideoCardz, “NVIDIA RTX 6000 (Ada) workstation GPU with 48GB memory is now available”: https://videocardz.com/newz/nvidia-rtx-6000-ada-workstation-gpu-with-48gb-memory-is-now-available-price-starts-at-7350 — RTX 6000 Ada MSRP.
4. Thunder Compute, “NVIDIA RTX A6000 Pricing (August 2026)”: https://www.thundercompute.com/blog/nvidia-rtx-a6000-pricing — RTX A6000 hardware and rental pricing.
5. Apple Newsroom, “Apple introduces new Mac Studio with M5 Max and M5 Ultra” (August 25, 2026): https://www.apple.com/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ — official max unified memory (512 GB), memory bandwidth (1.2 TB/s), and base pricing ($5,499).
6. MacRumors, “Mac Studio With M5 Ultra Chip and 512GB of RAM Launching in October”: https://www.macrumors.com/2026/08/25/mac-studio-m5-ultra-512gb-ram-october/ — 512GB configuration delayed, price unannounced.
7. AppleInsider, “You can spend $18,299 on a Mac Studio today, or more in October”: https://appleinsider.com/articles/26/08/25/you-can-spend-18299-on-a-mac-studio-today-or-more-in-october — current maximum orderable configuration and price.
8. NVIDIA, “NVIDIA H100 Tensor Core GPU” datasheet (via reseller/comparison aggregation of NVIDIA-published specs): 80 GB HBM3, 3.35 TB/s memory bandwidth.
9. NVIDIA, “NVIDIA A100 Tensor Core GPU” datasheet (via reseller/comparison aggregation of NVIDIA-published specs): 80 GB HBM2e, ~2 TB/s memory bandwidth.
10. U.S. Energy Information Administration, Electric Power Monthly, Table 5.6.A: https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a — U.S. average residential (18.34¢/kWh) and commercial (14.19¢/kWh) retail electricity prices, June 2026 data, published August 26, 2026.
11. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,” arXiv:2305.13245: https://arxiv.org/abs/2305.13245
12. NousResearch/Meta-Llama-3-70B-Instruct, config.json on Hugging Face: https://huggingface.co/NousResearch/Meta-Llama-3-70B-Instruct/blob/main/config.json — 80 layers, 64 attention heads, 8 key/value heads (GQA), head dimension 128.
13. Kurtic, E., Marques, A., Pandit, S., Kurtz, M., & Alistarh, D., “‘Give Me BF16 or Give Me Death’? Accuracy-Performance Trade-Offs in LLM Quantization,” ACL 2025, arXiv:2411.02355: https://arxiv.org/abs/2411.02355
14. Li, Z., Su, Y., Wang, S., Yang, R., Xie, C., Liu, A., Li, M., Cao, J., Xie, Y., Wong, N., & Yang, H., “Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning,” arXiv:2505.11574: https://arxiv.org/abs/2505.11574
15. RunPod, “H100 SXM GPU Rental” and “RTX 4090 GPU Rental”: https://www.runpod.io/gpu-models/h100-sxm , https://www.runpod.io/gpu-models/rtx-4090 — Community/Secure Cloud hourly rates.
16. Vast.ai marketplace pricing aggregation, 2026 — RTX 4090 typical on-demand range.
17. Lambda, Cloud GPU pricing pages — H100 SXM and A100 80GB hourly rates, 2026.
18. vLLM Blog, “vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction”: https://vllm.ai/blog/2024-09-05-perf-update — continuous-batching throughput gains from serving-engine improvements alone.
19. NVIDIA, MLPerf Inference v4.1 submission coverage (blogs.nvidia.com and independent MLPerf reporting) — H100 Llama-2-70B offline batched throughput benchmark.
20. This wiki, “AI subscription and pricing models in 2026”: /comparisons/ai-subscription-pricing-2026/ — consumer subscription tier pricing baseline, June 2026.
21. Anthropic, API pricing: https://platform.claude.com/docs/en/docs/about-claude/pricing — Claude Sonnet 5 at $2/$10 per million input/output tokens.
22. This wiki, “On-premise vs cloud for AI workloads”: /comparisons/on-premise-vs-cloud-ai/ — on-prem hardware cost ranges and 3–4 year refresh cycle.

Further reading

• On-premise vs cloud for AI workloads : the general capital-vs-operating-expense argument this page applies specifically to LLM inference.
• GPU clouds and neoclouds compared : how the rented-GPU providers named in the hardware ladder differ beyond price.
• When automation pays for itself : the same continuous-vs-intermittent utilization logic, generalized beyond GPUs.
• Managed vs. reserved vs. self-hosted inference : where self-hosting sits against managed and reserved-capacity alternatives once you’re serving more than yourself.
• GPU scheduling for AI : how to actually keep a self-hosted GPU busy enough to hit the batched end of the cost range on this page.
• AI subscription and pricing models in 2026 : the consumer subscription figures used as the comparison baseline above.