Triton Inference Server
hosomaki.triton
A Docker Compose file for Triton Inference Server alone, generated from the catalog and checked by the safety inspector. No account needed.
Inference server from NVIDIA hosting models from several frameworks — TensorRT, PyTorch, ONNX Runtime, Python — behind one HTTP and gRPC endpoint, with dynamic batching and per-model metrics.
Triton reads a model repository in which each model has its own versioned directory and configuration file, loading a backend appropriate to each format so one server can answer for a TensorRT engine, a TorchScript module and an ONNX graph at once. Dynamic batching groups arriving requests to raise device utilisation, concurrent model instances let several copies share a GPU, and ensembles chain models into a pipeline resolved server-side rather than by the caller. Load, latency and queue time are exported per model in Prometheus format. A first deployment mounts a repository containing one model and confirms the server reports it ready.
You know it worked when
- The server reports ready on its health endpoint.
- The repository index lists each expected model in a ready state.
- An inference request over HTTP returns output tensors of the declared shape.
- The same request over gRPC returns an identical result.
- The metrics endpoint reports request counts and queue times per model.
Known sharp edges
- The image is pulled from NVIDIA's NGC registry, which may require an account and accepted terms before the pull succeeds.
- The model repository layout is strict, and a misnamed directory or missing configuration file causes the model to be skipped rather than reported as an error.
- The image is several gigabytes compressed and first pull time dominates setup.
- Batching and instance counts are set in each model's configuration, not by the server, so defaults leave most device capacity unused.