Infinity
futomaki.infinity
A Docker Compose file for Infinity alone, generated from the catalog and checked by the safety inspector. No account needed.
Inference server for embedding, reranking and CLIP models, loading checkpoints from Hugging Face and answering over an OpenAI-compatible API with dynamic batching across concurrent requests.
Infinity runs sentence-transformer, cross-encoder reranker, CLIP and ColPali checkpoints behind a REST API, batching arriving requests dynamically so a device stays busy instead of idling between single calls. Several models can be held in one process and addressed by id, which lets an embedding model and a reranker share a GPU rather than each requiring its own. Backends include PyTorch, ONNX and TensorRT through optimum, and CTranslate2, chosen per deployment. A first deployment launches with one model id and confirms that an embedding request returns vectors of the expected dimensionality.
You know it worked when
- The server reports ready and its API documentation loads.
- The models endpoint lists each loaded model id.
- An embedding request returns a vector of the checkpoint's stated dimensionality.
- A rerank request returns scored documents in an order different from the input.
- Concurrent embedding requests are batched rather than answered strictly one at a time.
Known sharp edges
- Checkpoints download from Hugging Face on first start, and gated ones need a token supplied to the container.
- The default image targets CUDA; CPU-only hosts need the CPU image variant, and the ROCm and TensorRT variants are built outside CI and should be pinned to an exact version.
- Loading several models at once multiplies device memory use and there is no admission control to prevent an oversubscribed device failing mid-request.
- Embedding output is specific to the exact checkpoint, so changing the model id invalidates every vector already stored downstream.