vLLM
hosomaki.vllm
A Docker Compose file for vLLM alone, generated from the catalog and checked by the safety inspector. No account needed.
Inference engine for large language models built around PagedAttention, which pages the attention cache to raise batch sizes and throughput. Exposes an OpenAI-compatible API.
vLLM holds the key-value cache in fixed-size pages rather than one contiguous block per sequence, removing most of the memory fragmentation that limits how many requests can share a GPU. Requests join and leave the running batch as they arrive and finish, and prefixes shared between prompts are reused rather than recomputed. The server speaks the OpenAI completions and chat completions formats, so existing clients need only a changed base URL and key. A first deployment starts the container with one model id and a GPU, then issues a chat completion against it.
You know it worked when
- The server answers its health endpoint and lists the loaded model.
- A chat completion request in OpenAI format returns generated text.
- A streaming request returns tokens incrementally.
- Concurrent requests are answered in parallel rather than serialising behind one another.
- GPU memory use holds steady across sustained load rather than climbing until failure.
Known sharp edges
- A GPU with matching drivers and container toolkit is required; the server exits at startup without one.
- A large share of GPU memory is claimed for the cache at launch, so two engines cannot share a device without lowering the utilisation fraction on each.
- Weights download on first start and are not retained across container recreation unless the cache directory is a persistent volume.
- Support for quantised and unusual architectures is uneven, and an unsupported checkpoint fails at load rather than running slowly.