Every model OrionPod runs is a local server with an OpenAI- and Anthropic-compatible API. Drive it from a terminal, put your coding agent on it, and see exactly what the model was sent and what it said.
Every model runs as its own server. Start instances from the app or a terminal, put them all behind one gateway, and watch each one work.
The Instances page shows every running model server, whether the app or a terminal started it: health, port, uptime, requests, speed, memory and logs.
The gateway on :8400 routes each request by its model, starts an instance when one is needed and stops idle ones.
Quit the app and choose whether instances stop or keep serving. An engine that crashes is restarted on its own.
Share an instance with other devices: it announces itself over the LAN, needs an API key and can serve HTTPS. No account involved.
Context length, KV-cache precision, flash attention, speculative decoding, MoE offload and GPU layers, per instance, per model or as defaults.
An instance serves requests in parallel and reuses the cached prompt across turns, so agents and other clients don't queue behind each other.
The orionpod command ships inside the app. Pull a model, serve it, point an agent at it, and see what's running.
$ orionpod serve --model qwen3-coder OrionPod serving on http://127.0.0.1:8420 OpenAI base URL: http://127.0.0.1:8420/v1 Anthropic base URL: http://127.0.0.1:8420 Model: qwen3-coder (loading) Metrics: http://127.0.0.1:8420/metrics Log: ~/.orionpod/logs/instances/serve-48213.log Press Ctrl-C to stop.
Example output. Speeds and sizes depend on your model and hardware.
It comes with the app. Instances → Install command line tool puts orionpod on your PATH, once.
What you start in a terminal shows up in the app, and the other way round. orionpod stop stops one by port, pid or model.
list, rm, doctor, lan, requests, network, mcp and plugins, with --json where a script would want it.
Every instance speaks the OpenAI and Anthropic APIs. Change one base URL and your coding agent, SDK or script runs on a model on your machine.
One command starts the agent wired to a local instance, and starts one sized for agents if none is running.
orionpod check
Eight checks tell you whether a model can do agent work before you hand it your repo. Results show as a badge on the model.
from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8420/v1", api_key="unused", ) reply = client.chat.completions.create( model="llama-3.2-1b", messages=[{"role": "user", "content": "Hello!"}], ) print(reply.choices[0].message.content)
curl http://127.0.0.1:8420/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama-3.2-1b", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }'
from anthropic import Anthropic client = Anthropic( base_url="http://127.0.0.1:8420", api_key="unused", ) message = client.messages.create( model="llama-3.2-1b", max_tokens=256, messages=[{"role": "user", "content": "Hello!"}], ) print(message.content[0].text)
# One port for every model: the gateway picks the instance # from the request's model and starts it if it isn't running. client = OpenAI( base_url="http://127.0.0.1:8400/v1", api_key="unused", ) client.chat.completions.create(model="qwen3-coder", messages=messages) client.chat.completions.create(model="llama-3.2-1b", messages=messages)
Every request an instance serves is on record, with its timings, tokens and cache hits. Open one to read the prompt exactly as the model got it, and how sure it was of every token.
<|im_start|>system You are a concise assistant. Today is Thu 12 Nov.<|im_end|> <|im_start|>user What is the capital of Australia?<|im_end|> <|im_start|>assistant
This request continues the last turn, but something early in the prompt changed, so 1,912 tokens were read again.
Hints also catch a tool call the parser didn't recognise, reasoning that never closed and a prompt that was cut to fit.
Route, status, duration, first-token time, tokens, speed, cache hits and bodies. Search, filter and sort them.
Prometheus /metrics on every instance, shown as searchable cards in the app, and optional OpenTelemetry traces.
Live instance and engine logs, colour-coded by level and source, with search, filters and sorting.
Send a request again, to this instance or another model, and see both side by side. Save real traffic as a session to replay later.
Make a smaller quant of a model, measure what it cost and put the results side by side, all with llama.cpp's own tools. On macOS first.
Jobs check the model fits in memory and on disk first, show their progress, wait their turn and keep running if you close the app. Start them from the Jobs page or any terminal.
| qwen3-f16 | qwen3-q8 | qwen3-q4 | |
|---|---|---|---|
| Size | 15.2 GB | 8.1 GB | 4.7 GB |
| Bits per weight | 16.00 | 8.50 | 4.92 |
| Generate | 18.4 tok/s | 31.0 tok/s | 44.6 tok/s |
| Perplexity | 6.38 | 6.41 | 6.59 |
| KL divergence | base | 0.0021 | 0.0312 |
| Same top token | base | 99.1% | 94.6% |
| Agent checks | 8 / 8 | 8 / 8 | 7 / 8 |
Each model's report card, side by side, with the best value in every row marked. Charts follow speed, perplexity and drift across engine updates, and it all exports as Markdown or JSON.
A sensitivity sweep lowers one group of tensors at a time, keeps the rest byte for byte and measures what it costs, so you know where a model can least afford to lose precision.
The GGUF inspector shows all of a file's metadata, every tensor with its type and size, the tokenizer and how two files differ. Edit a chat template or any key into a new model.
Example data.
From Q8_0 down to IQ1_S, with an importance matrix from your own text if you like. The quant is saved as a new model that remembers what it was made from.
Perplexity on your own text, and KL divergence to see how far a quant drifts from its base model, token by token.
Read any GGUF file's header, diff two files and fix a model's metadata without touching its weights.
Load LoRA adapters and control vectors with an instance, and change an adapter's scale while it runs.
The agent engine inside OrionPod is now its own project. Orion Agent Harness is a backend-agnostic agent harness for local LLMs so anyone can build on it.
Bring your own model runtime — llama.cpp, MLX, a cloud API, anything — and orion-core handles everything above it: the conversation loop, context management and token budgets, streaming events, chat templates, and an automatic tool-execution loop.
Serving and building with OrionPod
No. The orionpod command ships with the app, and orionpod serve --model <name> starts an instance from any terminal. Instances the app started can keep serving after you quit it, and everything running shows up on the Instances page and in orionpod ps, whoever started it.
Yes. Every instance speaks the OpenAI and Anthropic APIs, so orionpod launch claude (or codex, opencode) starts the agent wired to a local model. orionpod check tests a model on tool calls, JSON output and long context first, so you know whether it is up to agent work.
Yes. Bind an instance to the network and it announces itself to other OrionPod apps nearby, with no account involved. An instance reachable beyond your own machine needs an API key before it starts, and HTTPS is available with your own certificate.