How to Serve an LLM with an OpenAI-Compatible API on a Dedicated GPU Server

in #technology • 13 days ago (edited)

How to Serve an LLM with an OpenAI-Compatible API

Running your own large language model is no longer limited to big tech companies. With a dedicated GPU server and an open-source engine like vLLM, you can deploy an open-weight model and expose it through an OpenAI-compatible API, so your applications can connect to it using familiar tools.

This post summarizes our latest tutorial from Leo Servers.

Why self-host an LLM?

  • Greater control over deployment and inference resources
  • Dedicated hardware instead of depending entirely on third-party inference
  • Easy integration using standard OpenAI-style requests and SDKs
  • Useful for chatbots, internal assistants, and document-processing systems

What the tutorial covers

  1. Concepts: how OpenAI-compatible serving works, including the /v1/chat/completions, /v1/completions and /v1/models endpoints
  2. Requirements: GPU VRAM, CPU, RAM, NVMe storage and Ubuntu 22.04/24.04
  3. Installation: creating an isolated Python 3.12 environment with uv and installing vLLM
  4. Model setup: using Qwen/Qwen2.5-7B-Instruct as an example and checking VRAM requirements first
  5. Deployment: launching the API server with an API key, bound to 127.0.0.1
  6. Testing: curl requests and a Python script using the OpenAI SDK
  7. Production: Nginx reverse proxy, HTTPS and firewall rules
  8. Monitoring: GPU utilization, memory, temperature, power, latency and throughput

Quick start

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Security reminder

Don't rely only on the --api-key option. It doesn't cover every endpoint on the HTTP server, so keep vLLM on localhost and protect access at the proxy and firewall level. Never expose an unrestricted inference endpoint to the public internet.


Read the full step-by-step guide:
How to Serve an LLM with an OpenAI-Compatible API on a GPU Server

Have you self-hosted an LLM? Tell me which inference engine you use in the comments.
serve-llm-openai-compatible-api-gpu-server.png

Sort:  

Buenísimo el detalle de que el --api-key no cubre todos los endpoints, eso es justo lo que muchos pasan por alto y terminan exponiendo el server. Con Qwen2.5-7B y vLLM en un dedicado andás sobrado para probar.

Using vLLM with an isolated environment via uv provides a clean deployment path. Binding to localhost and relying on an Nginx reverse proxy is definitely the right security approach for production.