Text Generation Inference
hosomaki.tgi
A Docker Compose file for Text Generation Inference alone, generated from the catalog and checked by the safety inspector. No account needed.
Inference server from Hugging Face for text generation models, using tensor parallelism, continuous batching and paged attention to hold throughput steady as concurrency rises.
TGI loads a transformer checkpoint across one or more GPUs and exposes it over HTTP with token streaming, returning partial output as it is produced rather than at the end of a generation. Continuous batching merges arriving requests into the running batch instead of waiting for a fixed window, which is what keeps per-request latency stable under load. Quantisation, tensor parallelism across devices and grammar-guided output are set at launch rather than per request. A first deployment starts the container with one model id and a GPU device, waits for the weights to download and shard, then streams a completion.
You know it worked when
- The server reports ready and its info endpoint names the loaded model.
- A generation request returns text.
- A streaming request delivers tokens incrementally rather than in one block.
- Several concurrent requests are answered without per-request latency collapsing.
- GPU memory use settles at a stable level after the first few requests.
Known sharp edges
- A GPU with matching host drivers and container toolkit is required; the server does not run usefully on CPU.
- Weights download and shard on first start, which can take a long time for large checkpoints and repeats on every container recreation unless the cache is a persistent volume.
- GPU memory is claimed up front for the attention cache, so an oversized batch or input-length setting fails at launch rather than degrading under load.
- Gated checkpoints need a Hugging Face token supplied to the container or the download stops with an authorisation error.