Ollama

By Ollama

Run open LLMs locally with one command — an OpenAI-compatible API for offline, private agents.

Website ↗ Repository ↗ Docs ↗

v0.32.12026-07-16GoMIT

#local #open-models #self-host #runtime

176.7k Open source Free tier Paid

Doc version 1.0 · Updated June 28, 2026

Related tools

LiteLLM

Gateways & Routers

Overview

Ollama runs open-weight models (Llama, Qwen, Gemma, …) on your own machine with a single command, exposing an OpenAI-compatible API on localhost:11434.
It is the local, no-key option for agents: free, private, and offline — and it drops into a LiteLLM-routed app by changing only the model string.

flowchart LR
  app[your app] --> srv[Ollama server]
  srv --> model[(local model)]
  model --> reply[reply]

The Code samples tab shows the CLI and the LiteLLM route — pick from the selector to compare.

When to use it

Reach for Ollama when you want to develop offline, keep data on-device, or avoid API costs — and use the ollama_chat/ prefix through LiteLLM when the agent needs tool calling.

Example

Shell · pull & run

When to use

Reach for the CLI to try a model interactively or pull it before a run.

How it flows

ollama downloads the model once, then serves it locally on port 11434.

Structure

flowchart LR
  cli[ollama run] --> srv[Ollama server]
  srv --> model[(local model)]

# Pull a model and chat locally — no API key, runs offline.
ollama pull qwen3.5:9b
ollama run qwen3.5:9b "Summarize Ollama in one line."

When to use

Route Ollama through LiteLLM so the same agent code runs on a local or a cloud model.

How it flows

ollama_chat/<model> calls the local server; use the chat endpoint (ollama_chat/, not ollama/) for tool calling.

Structure

flowchart LR
  app[your app] --> lite[LiteLLM]
  lite -->|ollama_chat/| srv[Ollama server]

import os
from litellm import completion

# ollama_chat/ uses Ollama's /api/chat endpoint, which supports tool calling.
resp = completion(
    model="ollama_chat/qwen3.5:9b",
    messages=[{"role": "user", "content": "Summarize Ollama in one line."}],
    api_base=os.environ.get("OLLAMA_API_BASE", "http://localhost:11434"),
)
print(resp.choices[0].message.content)

In a devcontainer with DooD, set OLLAMA_API_BASE=http://host.docker.internal:11434 to reach the host's Ollama.

Example

Ollama — a local LLM call via LiteLLM

A real, runnable mini-project. Download it and run with Docker.

View samples/ollama_1 on GitHub ↗

June 28, 2026

Ollama — a local LLM call via LiteLLM

A tiny script that calls a model running in a local Ollama server through LiteLLM — no API key, no cloud. Because it goes through LiteLLM, the same code runs against a cloud model by changing MODEL.

Configure

cd samples/ollama_1
cp .env.sample .env

First pull the model on the host — ollama pull qwen3.5:9b (or ollama run qwen3.5:9b). No API key is needed. .env sets MODEL and OLLAMA_API_BASE; in a devcontainer with DooD the container reaches the host's Ollama at http://host.docker.internal:11434 (already the default in .env.sample).

Tool calling needs the ollama_chat/ prefix (Ollama's chat endpoint), not ollama/ — and a tool-capable model.

Run with Docker

cd samples/ollama_1
docker build -t aas-ollama .
docker run --rm --env-file .env aas-ollama "Summarize Ollama in one line."

Run with Docker (in a devcontainer with DooD)

In a dev container that talks to the host Docker daemon (Docker-outside-of-Docker), the foreground docker run above often prints nothing and exits 0 — but the run itself succeeds. The script runs to completion and Docker captures all of its output; only the live attached stream drops it over the VM boundary. Run detached and follow the logs instead:

cd samples/ollama_1
docker build -t aas-ollama .
docker logs -f "$(docker run -d --env-file .env aas-ollama \
  "Summarize Ollama in one line.")"

Run locally

cd samples/ollama_1
pip install -r requirements.txt
# point at your local Ollama (no Docker): http://localhost:11434
OLLAMA_API_BASE=http://localhost:11434 python app.py "Summarize Ollama in one line."

Example run

Output varies by model and run — LLMs are non-deterministic, so the wording differs each time. Below is one run with ollama_chat/qwen3.5:9b.

Ollama allows users to easily run open-source large language models locally on
their own devices, enabling private and offline access to AI without relying on
external servers or internet connectivity.

Files

.env.sample

# Copy this file to `.env`.
#   cp .env.sample .env
#
# No API key needed — Ollama runs locally. Pull the model first on the host:
#   ollama pull qwen3.5:9b
# Use a tool-capable model and the ollama_chat/ prefix (chat endpoint).
MODEL=ollama_chat/qwen3.5:9b

# Where the Ollama server runs:
#   Local, no Docker:           http://localhost:11434
#   Docker / DooD on the host:  http://host.docker.internal:11434
OLLAMA_API_BASE=http://host.docker.internal:11434

app.py

"""Ollama: a local LLM call routed through LiteLLM.

Calls a model running in a local Ollama server — no API key. Routing through
LiteLLM means the same code also works against cloud models by changing MODEL.

Run (with Ollama running on the host):
    docker build -t aas-ollama .
    docker run --rm --env-file .env aas-ollama "Summarize Ollama in one line."
"""

import os
import sys

from dotenv import load_dotenv
from litellm import completion

# Load .env when running locally; in Docker the config comes from --env-file.
load_dotenv()

# ollama_chat/ uses Ollama's /api/chat endpoint (supports tool calling).
MODEL = os.environ.get("MODEL", "ollama_chat/qwen3.5:9b")


def main() -> None:
    prompt = " ".join(sys.argv[1:]) or "Summarize what Ollama does in one sentence."

    resp = completion(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        api_base=os.environ.get("OLLAMA_API_BASE", "http://localhost:11434"),
    )
    print(resp.choices[0].message.content)


if __name__ == "__main__":
    main()

Dockerfile

# Minimal image for the Ollama sample.
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

# Provide your config at run time (.env is NOT baked into the image):
#   docker run --rm --env-file .env aas-ollama "<prompt>"
ENTRYPOINT ["python", "app.py"]

Related tools

Overview

When to use it

Pricing

When to use

How it flows

When to use

How it flows

Configure

Run with Docker

Run with Docker (in a devcontainer with DooD)

Run locally

Example run