BentoML
chirashi.bentoml
Python framework for turning trained models into HTTP inference endpoints, handling request batching, multi-model composition, dependency capture and a versioned, runnable build artifact.
A Docker Compose file for BentoML alone, generated from the catalog and checked by the safety inspector. No account needed.
Python framework for turning trained models into HTTP inference endpoints, handling request batching, multi-model composition, dependency capture and a versioned, runnable build artifact.
BentoML wraps inference code as a service class with typed endpoints, so a trained model becomes an HTTP API without a separate web framework layer written around it. Adaptive batching groups concurrent requests at the runtime level to raise hardware utilisation, and several models can be composed into one service with independent scaling characteristics. The build step captures code, weights, Python dependencies and configuration into a versioned artifact that containerises reproducibly, which closes the gap between what ran in training and what runs in production. A first deployment defines one service, serves it locally and calls its generated endpoint.
You know it worked when
- The server starts and its generated API documentation lists the defined endpoints.
- An inference request returns a prediction in the declared output type.
- Concurrent requests are batched, visible in the server's own metrics.
- A build produces a versioned artifact that runs on a fresh container.
- A second model composed into the service answers on its own endpoint.
Known sharp edges
- The framework expects a service definition in code; pointing the container at a directory without one yields a server with no endpoints rather than an error.
- Dependency capture is only as accurate as the declared requirements, and an undeclared import fails at request time rather than at build time.
- Adaptive batching changes latency characteristics under load, and its window needs tuning against the actual request pattern.
- GPU inference requires the device passed through and matching runtime libraries inside the built artifact, neither of which the build verifies.