LiteLLM
hosomaki.litellm
A Docker Compose file for LiteLLM alone, generated from the catalog and checked by the safety inspector. No account needed.
Gateway fronting more than 100 model providers behind a single OpenAI-compatible API, adding routing rules, fallback chains, retries, per-key spend limits and request logging.
LiteLLM normalises the request and response formats of hosted and local model providers so that application code written against the OpenAI client can reach any of them unchanged. Deployed as a proxy it becomes the single endpoint applications call, holding provider credentials centrally and issuing virtual keys that carry their own budgets, rate limits and model allow-lists. Routing rules spread traffic across deployments of the same model and fall through to a second provider when one fails or rate-limits. A first deployment registers two model entries in the proxy configuration and issues a virtual key scoped to them.
You know it worked when
- The proxy reports healthy and lists the configured models at its models endpoint.
- A request in OpenAI format returns a completion from the first configured provider.
- The same request routed to a second provider returns a response in the same shape.
- A forced failure on the primary provider results in the fallback answering.
- Spend for that request appears against the virtual key that issued it.
Known sharp edges
- Provider credentials must exist in the proxy configuration before a route resolves; a missing key surfaces as an upstream authentication error rather than a gateway error.
- Routing to hosted providers spends real money, and without per-key budgets one misbehaving client can run up cost quickly.
- Virtual keys, budgets and spend history need an attached database; without one that state is lost on restart.
- The proxy holds every provider credential in one place, which makes its own authentication the single point that matters.