Self-hosting vLLM: Hardware costs, VRAM sizing, and GPU depreciation explained

Why vLLM changes the economics of LLM serving

vLLM, developed at UC Berkeley and first published in 2023, introduced PagedAttention — a memory management algorithm that treats GPU VRAM like virtual memory in operating systems. Instead of reserving contiguous blocks of VRAM for the KV cache of each concurrent request, PagedAttention stores KV cache blocks in non-contiguous pages that are allocated on demand and shared across requests using copy-on-write semantics.

The practical result: vLLM achieves 10–24× higher throughput than naive Hugging Face Transformers inference by enabling continuous batching, where the GPU processes tokens from multiple requests simultaneously rather than waiting for each request to complete before starting the next.

For broader AI infrastructure cost strategy, see our sovereign tech and AI computing cost framework.

---

VRAM sizing formula

The VRAM requirement for deploying a model with vLLM:

$$\text{VRAM}{\text{total}} = \text{VRAM}{\text{weights}} + \text{VRAM}_{\text{KV-cache}}$$

$$\text{VRAM}{\text{weights}} = \frac{N{\text{params}} \times P_{\text{bytes}}}{10^9}$$

Where $P_{\text{bytes}}$ = 2 for BF16, 1 for INT8, 0.5 for INT4 (GPTQ/AWQ quantization).

$$\text{VRAM}{\text{KV-cache}} \approx \frac{2 \times N{\text{heads}} \times d_{\text{head}} \times N_{\text{layers}} \times S_{\text{seq}} \times B_{\text{batch}}}{10^9} \times P_{\text{bytes}}$$

Practical VRAM requirements:

---

GPU depreciation model

GPU hardware depreciates on a non-linear curve driven by new generation releases and secondary market dynamics:

Depreciation-adjusted monthly cost (2× A100 80GB, $31,000 purchase, 24-month amortization):

• Average resale value at 24 months: ~$14,000
• Net cost: $31,000 - $14,000 = $17,000
• Monthly depreciation: $17,000 / 24 = $708/month

This is meaningfully lower than the straight-line amortization model ($31,000 / 24 = $1,292/month) and should be used for accurate cost projections.

---

vLLM throughput benchmarks

On a 2× A100 80GB configuration running Llama 3.1 70B in INT4 quantization with vLLM continuous batching:

• Peak throughput: 450–600 tokens/second output
• p50 TTFT (Time to First Token): 180–350ms
• p95 TTFT: 800–1,200ms under high concurrency
• Concurrent requests supported: 20–50 (limited by KV cache allocation)

Use the Bandwidth Calculator alongside the LLM Token Cost Calculator to model total infrastructure requirements including network egress for your self-hosted deployment.