GPU Server Requirements for Open-Weight AI Models: VRAM, KV Cache, and Bandwidth Explained
Running Llama, Mistral, Qwen, DeepSeek, or gpt-oss on your own hardware is now realistic. The key is knowing what your GPU server must hold before you rent it.
The four numbers that matter
1. Weight memory
Parameters × bytes per parameter. A 70B model needs about 140 GB at 16-bit, about 70 GB at 8-bit, and about 35 to 40 GB at 4-bit.
2. KV cache
This grows with context length and the number of active users. On Llama 3.1 70B, one 128,000-token conversation needs about 40 GB of cache by itself.
3. Memory bandwidth
Single-stream speed is roughly memory bandwidth divided by the bytes read per token. An H100 SXM has 3.35 TB/s, and an H200 has 4.8 TB/s with 141 GB of HBM3e.
4. Interconnect speed
NVLink offers up to 900 GB/s per GPU, while PCIe Gen5 x16 offers about 64 GB/s each way. This matters once a model spans four or more GPUs.
Don't forget
- Mixture-of-experts models still need every expert in memory
- Full fine-tuning needs about 16 bytes per parameter, while LoRA and QLoRA need far less
- System RAM, NVMe storage, power, and cooling all affect real-world performance
Read more
The full guide has a model-to-GPU matching table, fine-tuning requirements, and a 7-step planning checklist:
👉 https://www.leoservers.com/blogs/gpu-server-requirements-open-weight-ai/
Originally published at Leo Servers.
El dato del KV cache me voló la cabeza: 40 GB solo para una conversación de 128k tokens en Llama 3.1 70B, ahí te das cuenta que el VRAM se va más rápido de lo que uno planea. ¿La tabla del guide incluye casos con múltiples usuarios concurrentes o solo single-stream?