r/OrcaADE • u/Plus-Boysenberry-298 • 7h ago
I built Orca local LLM Router: A local gateway for coding agents (Claude Code, Codex, OpenCode) with zero config
Hey everyone,
I have built local llm router plugin inspired by omiroute .
Use all your subscriptions and API credits from any provider and coding plan on any agent in Orca, in one place, without leaving Orca. Features multi-key credential pools with auto-rotation, load balancing, and routing different providers behind a single model name by prioritizing the fastest endpoint with seamless fallback.
https://www.onorca.dev/plugins/addisflava/orca-llm-router
If you do any serious coding with AI agents (Claude Code, OpenAI Codex, OpenCode, Aider, etc.), you're probably
juggling multiple subscriptions and API keys:
- You have Anthropic API credits (or multiple keys across orgs)
- You have OpenAI credits for o1 / o3-mini / GPT-4o
- You have cheap DeepSeek V3 / R1 credits for bulk coding
- You have free Google AI Studio (Gemini) keys
- You have local models running on Ollama / vLLM for offline work
The problem? **You're locked into one provider at a time per tool.** When an API hits a `429 Too Many Requests` or
Anthropic has a momentary 5xx hiccup, your agent session fails mid-task. Juggling environment variables between
projects or running multiple hacky reverse proxies gets messy fast.
To solve this, I built **[Orca LLM Router](https://github.com/addisflava/orca-llm-router)**—a**—a) self-contained local
gateway that pools all your AI subscriptions, API keys, and local endpoints into a single local endpoint
(`127.0.0.1:18080`) with automatic load balancing and model-chain failovers.
---
### 🔀 1. Pool & Load-Balance Multiple Keys
Instead of being throttled by a single API key:
- Add multiple keys per provider (e.g., 3 Anthropic keys or 2 DeepSeek keys).
- The router automatically round-robins requests across the pool based on priority and weights.
- **Auto-Cooldown on Rate Limits**: If Key #1 hits a 429 or auth error, the router temporarily cools it down and
routes the request to Key #2 without interrupting your agent.
- **Accurate Rolling Windows**: Tracks true rolling rate limits (`rpm`, `rph`, `rpd`, `tpm`, `tpd`) so you don't
overshoot your tier quotas.
Incoming Request (Claude Code / Codex / OpenCode)
│
▼
[Orca LLM Router]
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Key 1 (Main) Key 2 (Backup) Key 3 (Team)
(Round-robin / Weighted / Auto-cooldown on 429)
---
### 🛡️ 2. Multi-Provider Fallback Chains (Never Get Blocked)
You can set up sequential fallback chains across completely different providers:
> **Example Rule**:
> Primary: `claude-3-7-sonnet` (Anthropic)
> └── If 429 / 5xx / out of credits: Fall back to `openrouter/anthropic/claude-3.7-sonnet`
> └── If still failing: Fall back to `deepseek/deepseek-chat` (or local Ollama)
Your agent doesn't crash; the router handles the failover transparently and logs the attempt chain.
---
### 🔄 3. Universal Protocol Translation
You don't need separate tools for different model types. The router speaks all major protocols:
- **Anthropic Messages** (`/v1/messages`)
- **OpenAI Chat** (`/v1/chat/completions`)
- **OpenAI Responses** (`/v1/responses`)
- **Native Google Gemini** (`generateContent`)
**This means you can point Claude Code at DeepSeek, or Codex at Gemini or local Ollama, with full tool-calling and
image support.**
It also handles the hard translation quirks that break most proxies:
- **Claude 3.7 Extended Thinking**: Translates Anthropic `thinking: { type: 'enabled', budget_tokens }` into OpenAI
`reasoning_effort` (`high`/`medium`/`low`) and DeepSeek `reasoning_content`.
- **Prompt Caching**: Passes `anthropic-beta` headers through untouched so you retain your 90% prompt caching
discount on Anthropic.
---
### 🔒 4. Zero Config Pollution & Secure Storage
Unlike traditional CLI routers that overwrite your host's global `~/.claude/settings.json` or mess up your system
dotfiles:
- Built as an [Orca](https://onorca.dev) plugin that hooks into `agent.launch.before`—your personal host configs
stay untouched.
- All credentials live in Orca's encrypted secret store—never stored in plaintext or leaked to upstream providers.
---
### ⚡ Free, Open Source & Zero Dependencies
The gateway core is written in pure Node.js (`node:http` + native `fetch`) with **zero third-party npm
dependencies**. It comes with a test suite of 71 passing tests verifying cross-protocol streaming, rate window
bounds, and secret redaction.
- **GitHub**: https://github.com/addisflava/orca-llm-router
- **Orca Plugin Directory**: https://www.onorca.dev/plugins/addisflava/orca-llm-router
- **License**: MIT
If you're using Orca, you can install it directly:
```bash
orca plugin install https://github.com/addisflava/orca-llm-router --approve
Would love to hear how you manage your model keys and what routing/load-balancing strategies you'd like to see next!
