How to Serve an LLM with an OpenAI-Compatible API on a Dedicated GPU Server
Running your own large language model is no longer limited to big tech companies. With a dedicated GPU server and an open-source engine like vLLM, you can deploy an open-weight model and expose it through an OpenAI-compatible API, so your applications can connect to it using familiar tools.
This post summarizes our latest tutorial from Leo Servers.
Why self-host an LLM?
- Greater control over deployment and inference resources
- Dedicated hardware instead of depending entirely on third-party inference
- Easy integration using standard OpenAI-style requests and SDKs
- Useful for chatbots, internal assistants, and document-processing systems
What the tutorial covers
- Concepts: how OpenAI-compatible serving works, including the
/v1/chat/completions,/v1/completionsand/v1/modelsendpoints - Requirements: GPU VRAM, CPU, RAM, NVMe storage and Ubuntu 22.04/24.04
- Installation: creating an isolated Python 3.12 environment with
uvand installing vLLM - Model setup: using
Qwen/Qwen2.5-7B-Instructas an example and checking VRAM requirements first - Deployment: launching the API server with an API key, bound to
127.0.0.1 - Testing: curl requests and a Python script using the OpenAI SDK
- Production: Nginx reverse proxy, HTTPS and firewall rules
- Monitoring: GPU utilization, memory, temperature, power, latency and throughput
Quick start
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
Security reminder
Don't rely only on the --api-key option. It doesn't cover every endpoint on the HTTP server, so keep vLLM on localhost and protect access at the proxy and firewall level. Never expose an unrestricted inference endpoint to the public internet.
Read the full step-by-step guide:
How to Serve an LLM with an OpenAI-Compatible API on a GPU Server
Have you self-hosted an LLM? Tell me which inference engine you use in the comments.
Buenísimo el detalle de que el --api-key no cubre todos los endpoints, eso es justo lo que muchos pasan por alto y terminan exponiendo el server. Con Qwen2.5-7B y vLLM en un dedicado andás sobrado para probar.
Using vLLM with an isolated environment via uv provides a clean deployment path. Binding to localhost and relying on an Nginx reverse proxy is definitely the right security approach for production.