How to Deploy Gemma 3 on a Dedicated GPU Server with vLLM and Docker
Running AI on your own hardware is becoming the smart choice for teams that care about privacy, speed and predictable costs. In this post I share a complete guide to deploying Google’s Gemma 3 on a dedicated Ubuntu GPU server using vLLM and Docker.
What you will get
A Gemma 3 endpoint that is bound to localhost, managed by systemd and tested with both text and images. vLLM provides an OpenAI-compatible API, so your existing tools can connect easily.
Which model should you pick?
Gemma 3 4B: around one 16 GB GPU
Gemma 3 12B: one 40–48 GB GPU, or two 24 GB GPUs
Gemma 3 27B: one 80 GB GPU, or several GPUs
You need an NVIDIA GPU with compute capability 8.0 or higher (Ampere or newer) for bfloat16.
What the tutorial covers
Installing the NVIDIA driver
Setting up Docker and the NVIDIA Container Toolkit
Storing your Hugging Face token safely
Running vLLM and testing the API
Creating a systemd service
Scaling, hardening and troubleshooting
Self-hosting means your data stays in your infrastructure and you avoid surprise API bills.
Read more
👉 https://www.fitservers.com/tutorials/howto/deploy-gemma-3-gpu-server/
El dato de la compute capability 8.0 mínimo para bfloat16 está bueno, porque ahí te das cuenta si tu GPU vieja sirve o no antes de meterte en todo el setup. ¿El systemd service lo armaste con restart automático o lo dejás manual?