LiteLLM

By BerriAI

Call 100+ LLM APIs through one OpenAI-style interface, plus a proxy with routing, fallbacks, and cost tracking.

Website ↗ Repository ↗ Docs ↗

v1.93.02026-07-19PythonMIT

#gateway #multi-provider #proxy #python

54.4k Open source Paid

Doc version 1.0 · Updated June 28, 2026

Related concepts

Harness Engineering Doc version 1.0 · Updated June 28, 2026 Wrapping an LLM in scaffolding — a control loop, tools, a sandbox, guardrails, and evaluation — so it behaves reliably in production.

Overview

LiteLLM is a thin abstraction over many model providers (Anthropic, OpenAI, Bedrock, Vertex, local models, …) exposing a single OpenAI-compatible call.
Beyond the SDK, its proxy server acts as an LLM gateway: load balancing, automatic fallbacks, retries, budgets, and per-key cost tracking — so agents can swap models without code changes.

flowchart LR
  agent[agent code] -->|OpenAI-style call| litellm[LiteLLM]
  litellm --> claude[Anthropic]
  litellm --> openai[OpenAI]
  litellm --> gemini[Gemini]

The Code samples tab shows two ways to use LiteLLM — pick the SDK call or the Router fallback from the selector to compare.

When to use it

Use LiteLLM when an agent must stay provider-agnostic, needs fallbacks across models, or when you want a central gateway to enforce budgets and track spend.

Example

SDK · completion()

When to use

Reach for the SDK when you just want one provider-agnostic call. Switch providers by editing only the model string.

How it flows

LiteLLM translates the OpenAI-style request into the target provider's API and normalizes the response back.

Structure

flowchart LR
  app[your code] -->|completion| litellm[LiteLLM]
  litellm --> prov[(any provider)]

from litellm import completion

# One call shape for every provider — switch by editing only the model string.
resp = completion(
    model="anthropic/claude-opus-4-8",
    messages=[{"role": "user", "content": "Summarize litellm in one line."}],
)
print(resp.choices[0].message.content)

For a runnable build of this call, see litellm_1 in the Implementation tab.

When to use

Reach for the Router when availability matters. It falls back to another model when the primary fails, and load-balances across deployments.

How it flows

If the primary call fails, the Router follows the fallbacks map and retries on backup.

Structure

flowchart LR
  app[your code] --> router[Router]
  router -->|primary| A[claude-opus-4-8]
  router -.->|on error| B[gpt-4o]

from litellm import Router

# Try the primary model; if it errors, fall back to the next one.
router = Router(
    model_list=[
        {"model_name": "primary", "litellm_params": {"model": "anthropic/claude-opus-4-8"}},
        {"model_name": "backup",  "litellm_params": {"model": "openai/gpt-4o"}},
    ],
    fallbacks=[{"primary": ["backup"]}],
)

resp = router.completion(
    model="primary",
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)

Example

LiteLLM — one interface, any provider

A real, runnable mini-project. Download it and run with Docker.

View samples/litellm_1 on GitHub ↗

June 28, 2026

LiteLLM — one interface, any provider

A tiny LiteLLM script: it sends one prompt through litellm.completion() and prints the reply. The same call works with Anthropic Claude, OpenAI, or Google AI Studio (Gemini) — change MODEL in .env, never the code.

Configure

cd samples/litellm_1
cp .env.sample .env
# edit .env: set MODEL and the matching provider key

MODEL picks the provider:

Provider	`MODEL` example	Key in `.env`
Anthropic Claude	`claude-opus-4-8`	`ANTHROPIC_API_KEY`
OpenAI	`gpt-4o`	`OPENAI_API_KEY`
Google AI Studio	`gemini/gemini-2.5-flash`	`GEMINI_API_KEY`
Ollama (local)	`ollama_chat/qwen3.5:9b`	`OLLAMA_API_BASE`

.env is gitignored — only .env.sample is committed.

Ollama (local models): first pull the model on the host — ollama pull qwen3.5:9b (or ollama run qwen3.5:9b). Then set MODEL=ollama_chat/qwen3.5:9b and point OLLAMA_API_BASE at the server — no API key needed. In a devcontainer with DooD the container reaches the host's Ollama at http://host.docker.internal:11434; running locally use http://localhost:11434.

Run with Docker

cd samples/litellm_1
docker build -t aas-litellm .
docker run --rm --env-file .env aas-litellm "Summarize litellm in one line."

Run with Docker (in a devcontainer with DooD)

In a dev container that talks to the host Docker daemon (Docker-outside-of-Docker), the foreground docker run above often prints nothing and exits 0 — but the run itself succeeds. The script runs to completion and Docker captures all of its output; only the live attached stream drops it over the VM boundary. You can confirm this: docker logs on the same container shows the full output, the container exits 0, and it is not an OOM. Run detached and follow the logs instead:

cd samples/litellm_1
docker build -t aas-litellm .
docker logs -f "$(docker run -d --env-file .env aas-litellm \
  "Summarize litellm in one line.")"

Run locally

cd samples/litellm_1
pip install -r requirements.txt
python app.py "Summarize litellm in one line."

python-dotenv loads .env automatically. Get keys from Anthropic, OpenAI, or Google AI Studio.

Example run

Output varies by model and run — LLMs are non-deterministic, so the exact wording (and an agent's steps) differ each time. Below is one run with claude-opus-4-8.

LiteLLM is an open-source library that provides a unified interface to call 100+ LLM APIs (OpenAI, Anthropic, Azure, Cohere, etc.) using the OpenAI format.

Files

.env.sample

# Copy this file to `.env` and fill in the key for the provider you want.
#   cp .env.sample .env
#
# One codebase, many providers — LiteLLM routes the request based on MODEL.

# Pick the model. Examples:
#   claude-opus-4-8              Anthropic Claude   -> needs ANTHROPIC_API_KEY
#   gpt-4o                       OpenAI             -> needs OPENAI_API_KEY
#   gemini/gemini-2.5-flash      Google AI Studio   -> needs GEMINI_API_KEY
#   ollama_chat/qwen3.5:9b       Ollama (local)     -> needs OLLAMA_API_BASE, no key
MODEL=claude-opus-4-8

# Set only the key for the provider you chose; leave the rest blank.
ANTHROPIC_API_KEY=
OPENAI_API_KEY=
GEMINI_API_KEY=

# Ollama only — where the Ollama server runs (pull the model first, e.g.
# `ollama pull qwen3.5:9b`). Other providers ignore this line.
#   Local, no Docker:           http://localhost:11434
#   Docker / DooD on the host:  http://host.docker.internal:11434
OLLAMA_API_BASE=http://host.docker.internal:11434

app.py

"""LiteLLM: one provider-agnostic call for every model.

LiteLLM exposes 100+ providers behind a single OpenAI-style `completion()`
call. Pick the model with MODEL in `.env` (claude-opus-4-8, gpt-4o,
gemini/gemini-2.5-flash, …) and the calling code never changes — LiteLLM routes
the request to the right provider and normalizes the response back.

Run:
    docker build -t aas-litellm .
    docker run --rm --env-file .env aas-litellm "Summarize litellm in one line."
"""

import os
import sys

import litellm
from dotenv import load_dotenv

# Load .env when running locally; in Docker the keys come from --env-file.
load_dotenv()

# Drop params a given provider doesn't support, so one call works everywhere.
litellm.drop_params = True


DEFAULT_PROMPT = "Summarize what LiteLLM does in one sentence."


def main() -> None:
    prompt = " ".join(sys.argv[1:]) or DEFAULT_PROMPT

    # MODEL chooses the provider (claude-opus-4-8 / gpt-4o / gemini/gemini-2.5-flash).
    model = os.environ.get("MODEL", "claude-opus-4-8")

    resp = litellm.completion(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    print(resp.choices[0].message.content)


if __name__ == "__main__":
    main()

Dockerfile

# Minimal image for the LiteLLM sample.
FROM python:3.13-slim

# Stream stdout/stderr unbuffered so the output appears live.
ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

# Provide your config at run time (.env is NOT baked into the image):
#   docker run --rm --env-file .env aas-litellm "<prompt>"
ENTRYPOINT ["python", "app.py"]

Related concepts

Overview

When to use it

Pricing

When to use

How it flows

When to use

How it flows

Configure

Run with Docker

Run with Docker (in a devcontainer with DooD)

Run locally

Example run