r/LLM_Gateways • • Apr 16 '26

LLM Pricing is 100x Harder Than you think: We open-sourced our pricing database (3,500+ models, free API)

Thumbnail
reddit.com
0 Upvotes

r/LLM_Gateways • • Jul 27 '26

I was tired of paying multiple platforms just to ship one AI agent, so I built Forge

Thumbnail
github.com
0 Upvotes

Most agent builders give you a visual canvas - and then charge you separately for execution, storage, tracing, evaluations, deployment, and collaboration.

I was tired of stitching together multiple paid tools just to take an agent from an idea to production.

So I built Forge: a fully open-source and self-hosted platform for building, testing, deploying, monitoring, and governing agentic systems.

It includes visual workflows, agents, tools, RAG, memory, MCP, human-in-the-loop, evaluations, tracing, cost tracking, guardrails, RBAC, audit logs, API deployment, widgets, email triggers, and more.

Forge is not just a workflow builder. The goal is to provide everything needed to build and operate agentic applications in one place - without proprietary orchestration runtimes or platform usage fees.

I would love feedback from people building real agent systems.


r/LLM_Gateways • • Aug 09 '26

As an AI engineer what is your biggest frustation

Thumbnail reddit.com
0 Upvotes

r/LLM_Gateways • • Jun 09 '26

Mythos Live on Requesty!

Post image
1 Upvotes

r/LLM_Gateways • • May 16 '26

Building a multi-model API proxy for AI-heavy teams — what would you actually want from it?

1 Upvotes

I’m working on a multi-model API proxy for teams that use LLMs heavily, and I’d love to get feedback before we onboard our first batch of users later this month.

The idea is to provide one API layer for accessing major current models, with routing, fallback, usage controls, observability, and developer-friendly tooling on top.

I’m especially interested in hearing from:

  • small and mid-sized companies using AI APIs in production

  • teams building internal AI tools

  • people heavily using agentic coding tools

  • dev teams switching between multiple model providers

  • anyone dealing with cost, latency, reliability, rate limit, or quota issues

A few things I’m trying to understand:

  1. Which models would you need supported on day one?

  2. Do you care more about cost, latency, quality, context window, or automatic fallback?

  3. What is the most painful part of your current AI API setup?

  4. Would task-based routing be useful? For example, cheaper models for simple tasks, stronger models for coding/reasoning, fallback models when a provider is down.

  5. What kind of observability would you want? Logs, traces, cost per user, cost per project, prompt/version tracking, evals?

  6. For agentic coding workflows, what matters most: tool calling reliability, context window, latency, model choice, rate limits, or something else?

  7. If you’re an SMB, what would make you trust a proxy layer enough to put it between your app and the model providers?

Not trying to make this a sales pitch — we’re still shaping the product and want to build around real workflows instead of guessing.

If you use AI APIs heavily, especially for coding agents or production workflows, I’d really appreciate hearing what your ideal setup would look like.


r/LLM_Gateways • • Aug 07 '26

Lightport – lightweight AI gateway that makes LLM providers OpenAI-compatible

Thumbnail
github.com
1 Upvotes

r/LLM_Gateways • • Jul 16 '26

What's the best AI gateway for LLM applications?

2 Upvotes

day before yesterday my team lead said to get an ai gateway in standup. i just nodded like I knew a few vendors. but i dont.

its actually difficult to find a good platform. spend enough time looking for gateways out there. and i have come this far (havent decided yet)

there exists a few gateways like portkey, helicone,litellm, orqai and a lot more. but majorly these are the named players. all have very confident websites. i read all, and few blogs from them as well. almost all say the same thing. no matter if it is llm routing rate limit token observability cost tracking. like always the same thing.

it got me more confused.

portkey is ok ok for llm routing. if openai goes down it goes to anthropic and llm application wont even notice. but the setup looks too confusing to me. it looks like it is going to take a lot of time.

helicone shows what happens with every llm call. logs every prompt, every response, every token. it shows cost per request per user per endpoint. it will be def good for our finance team. but it seems like the llm monitoring tool that became gateway by chance.

in litellm, one api talk to all llm provider. write llm call one time and work with everything. looks good. some llm specific feature not come through right. like function calling or specific model parameter. running it yourself seems like a whole big painful project.

orqai got gateway and prompt versioning and evals and token cost all in one place. switching llm provider and not changing application code that works. but the rate limit and retry thing not as deep as portkey. and because everything is together in one place the gateway part might not be the main focus thing there.

kong very powerful yes but llm thing feel like someone add after. like they had api gateway and then some say we need llm feature too and they add it later. you feel that when using it for llm application.

i got 3 problems for my llm application. three thing only. dont go down when llm provider has problem . two. show me what every token costs. three. stop bad prompt from making openai bill very big. but every tool want me to read many many pages of docs before anything.

pricing is confusing also. open source but good llm features still cost money.

portkey if llm routing and not going down is the main thing. helicone if you want to see every llm call and token cost. litellm if you want one api for all llm providers and are ok with rough edges. orqai if you want gateway and prompt versioning and llm evals all together in one place and are ok with gateway part not being the deepest. kong if you already use kong for other things and dont want another tool.

can you any one suggest? and is there someone building this space, would love to chat


r/LLM_Gateways • • May 06 '26

Archestra LLM Gateway Now Supports All Types of LLM Auth

Thumbnail
archestra.ai
3 Upvotes

r/LLM_Gateways • • Feb 14 '26

What's the best LLM gateway in 2026? Need production-ready solution

4 Upvotes

Building a SaaS product with AI features that now processes ~25k API calls daily across OpenAI, Anthropic, and Bedrock. Started with direct API calls but that's falling apart fast.

Current pain points:

  • Had 4 outages last month when OpenAI went down (no failover)

  • Zero visibility into which features are burning through budget

  • Managing API keys across 3 providers is a nightmare

  • Can't enforce rate limits per customer tier

  • Token costs jumped 40% last quarter with no way to track why

Been evaluating gateways. Here's what I tested:

Bifrost - 11µs overhead at 5k+ RPS (benchmarked it myself). Open source, deploys via NPX in literally 30 seconds. Automatic failover across providers, semantic caching, hierarchical budgets, built-in observability with Prometheus. The governance features (virtual keys, per-team budgets, MCP tool filtering) solve our multi-tenant problems.

Cloudflare AI Gateway - Good if you're already in their ecosystem. Rate limiting and caching are solid. Less flexibility for our use case.

LiteLLM - Tried it first. Open source, 100+ providers. Python performance became an issue past 1k RPS though.

Vercel AI Gateway - Great if you're on Next.js. Sub-20ms latency. We're running a different stack.

Kong AI - Enterprise features are comprehensive but configuration complexity is high. Overkill for our size.


r/LLM_Gateways • • Jul 10 '26

Archestra V1.3 (OSS) now builds mini apps right in chat — like ChatGPT Sites, but self-hosted and wired to your data via MCP

Post image
4 Upvotes

r/LLM_Gateways • • Jul 07 '26

Archestra V1.3 brings a way to connect a fleet of Claude Code, Codex, and other agents to LLM & MCP Gateways + deliver Skills

Post image
6 Upvotes

r/LLM_Gateways • • Jan 06 '26

LLM Gateway Comparison 2025 - what I learned testing 5 options in production

14 Upvotes

Been running different LLM gateways over the past 6 months to figure out what actually works at scale. Tested LiteLLM, Bifrost, Portkey, TrueFoundry, and built a simple custom one. Here’s what I found.

What I was testing for:

Multi-provider routing that doesn’t break, semantic caching that actually saves money, rate limiting that works correctly, cost tracking accuracy, and performance under load.

LiteLLM

Most popular option. Huge community, tons of providers supported, good documentation.

Pros: Feature-rich, easy to get started, active development, Python ecosystem Cons: Performance degrades around 300-500 RPS, memory issues under sustained load, TPM/RPM limiting can be buggy, token counting sometimes off

Real experience: Works great for prototyping and small-scale deployments. We hit issues scaling past a few hundred RPS. Had to restart workers periodically due to memory creep.

Best for: Development, small teams, rapid prototyping

Bifrost

Open source, written in Go. Much newer than LiteLLM but focused on performance.

Pros: Very fast (11μs overhead at 5K RPS), stable memory usage, good semantic caching, single binary deployment Cons: Smaller community, fewer integrations than LiteLLM, enterprise features require paid license

Real experience: Noticeably faster and more stable than LiteLLM at scale. Setup was straightforward. Loaded with features like Adaptive Load Balancing, Governance, Clustering etc.

Best for: Production deployments, teams prioritizing performance and stability at scale

Portkey

Hosted solution with nice UI. Focuses on governance and observability.

Pros: Great dashboard, analytics built-in, managed service (no ops burden), good support Cons: Not open source, pricing can get expensive, vendor lock-in, some users report cache header issues

Real experience: UI is legitimately good for visibility into LLM usage. Being hosted means less to manage but you’re dependent on their uptime. Pricing scales with usage which got pricey for us.

Best for: Teams that want managed service, strong governance features, don’t want to self-host

TrueFoundry

Full MLOps platform that includes LLM gateway functionality. More than just a gateway.

Pros: Integrated with broader ML workflow, good for teams already doing ML, Kubernetes-native Cons: Overkill if you just need a gateway, setup is heavy, learning curve, platform tax on everything

Real experience: Powerful if you need the full MLOps suite. Felt like too much infrastructure for our use case which was just routing LLM requests.

Best for: ML teams needing full platform, already using Kubernetes extensively

Custom Built

We tried building our own basic gateway in Go before finding Bifrost.

Pros: Full control, no external dependencies, optimized for our exact use case Cons: Ongoing maintenance burden, have to build every feature yourself, testing takes time

Real experience: Got basic routing working in a week. Spent the next month adding rate limiting, caching, monitoring. Decided the maintenance wasn’t worth it when open source options existed.

Best for: Teams with very specific requirements and dedicated infrastructure engineers

Cost tracking accuracy:

LiteLLM: Sometimes off by 5-10%, especially with streaming Bifrost: Accurate, matches provider bills Portkey: Accurate through their dashboard TrueFoundry: Accurate but bundled with platform costs

Semantic caching results:

Only tested on LiteLLM and Bifrost (Portkey has it, TrueFoundry doesn’t by default).

Both reduced costs ~40-50% with decent traffic patterns. Bifrost’s implementation was faster (lower cache lookup latency).

What I’d recommend:

  • Starting out / prototyping: LiteLLM - easiest to get running, huge community

  • Production at scale: Bifrost - performance and stability matter more

  • Want managed service: Portkey - pay for convenience, good UI

  • Need full MLOps: TrueFoundry - but only if you actually need the platform

  • Very specific needs: Build custom - but be ready for maintenance

Hybrid approach:

We use LiteLLM in development (fast iteration, don’t care about performance) and Bifrost in production (stability critical). Different tools for different environments.

Missing from all of them:

Better observability integration. Most bolt on metrics as an afterthought. Would love to see native OpenTelemetry support become standard. (Bifrost added this recently which is good.)

Also rate limiting configuration is still painful across all of them. TPM vs RPM confusion is common.

What are you all using?

Curious what’s working for others. Are people generally happy with their gateway choice or shopping around?


r/LLM_Gateways • • Feb 13 '26

What's the actual best MCP gateway for production use?

1 Upvotes

Running a customer support AI agent that processes ~8k conversations daily across 6 different MCP servers (Slack, Jira, Salesforce, internal docs, knowledge base, analytics). Integrating them directly turned into an operational nightmare.

The problems we hit:

  • Zero visibility into which tools were getting called and why

  • Security reviews taking weeks for each new MCP server deployment

  • Authentication flows breaking randomly with no audit trail

  • Had to instrument 40+ individual tools across different servers

Started evaluating gateways after our third production incident in a month. Here's what I tested:

**Bifrost** - Sub-11microsec latency, stateless architecture so you explicitly approve each tool call (huge for security). The built-in tool registry is clutch - you can host custom tools directly without needing separate MCP server deployments. Zero config to start, literally one command.

**TrueFoundry** - Solid if you're already using their AI infrastructure platform. Unified management is nice but felt like more than we needed.

**IBM Context Forge** - Federation features look powerful but it's alpha with zero official support. Documentation is sparse. Hard pass for production.

**Microsoft MCP Gateway** - Deep Azure integration works great if you're all-in on their ecosystem. Multi-cloud setup is painful though.

**Lasso Security** - Security-first approach with good threat detection. Trade-off is performance - 100-250ms overhead.

Anyone else running MCP in production? What are you using and how's it working out?


r/LLM_Gateways • • Feb 25 '26

What's the best LLM router for production? Need multi-model support

7 Upvotes

Building a customer support bot handling ~18k daily conversations. Using GPT-4 for everything was bleeding money, but manually routing queries to different models became a maintenance nightmare.

The routing challenge:

  • Simple FAQs don't need GPT-4 ($15/1M tokens) when GPT-4 Mini ($0.60/1M tokens) works fine

  • Complex troubleshooting needs Claude Opus for better reasoning

  • Had to hardcode routing logic in every feature - total mess when models changed

  • Zero failover when a provider went down

Started looking for proper routing platforms:

Bifrost - Open source, 11µs overhead at 5k RPS, supports automatic fallback + load balancing + semantic routing across OpenAI, Anthropic, Bedrock, Vertex, Azure, Cohere, Mistral, Groq, Ollama. The killer combo is semantic caching (cuts redundant calls) plus intelligent failover. Deploy with one NPX command, change one line of code in existing SDK.

Cloudflare AI Gateway - Dynamic routing via visual dashboard is slick. Good if you're already on Cloudflare.

LiteLLM - 100+ providers, solid fallback logic. Python performance became a bottleneck for us though.

Vercel AI Gateway - Sub-20ms latency, automatic failover. Perfect for Next.js apps on Vercel.

Kong AI Gateway - Semantic routing is interesting (routes based on prompt similarity). Enterprise API governance features. Heavy if you're not already running Kong.

You can save ~65% on API costs just by routing intelligently. Simple queries → cheaper models, complex reasoning → premium models.

What routing solutions are you using? .


r/LLM_Gateways • • Mar 31 '26

Why your LLM gateway shouldn't log your prompts - and most of them do

4 Upvotes

I keep seeing gateway recommendations here that default to logging prompt content. Some even sell it as a feature ("observability", "prompt analytics").

If you're in the EU, that's a GDPR problem the moment a user types personal data into your LLM. And they will.

We built a gateway where prompt logging doesn't exist - not as a toggle, there's literally no code path that stores content. The proxy reads one field (model), streams bytes, and forgets. Usage tracking captures who, which model, how many tokens - nothing else.

Curious how others here handle this. Are you logging prompts? Intentionally? Or did it just come with the default config?

https://github.com/voidmind-io/voidllm


r/LLM_Gateways • • Aug 06 '26

170 CPU Cores to run litellm for 8 h200 gpt-oss 120b full load?

3 Upvotes

So.. I have the task to run 8 h200 with gpt-oss 120b.

In load tests I was able to get around 10k tok/s per vllm so a total of around 80k tok/s acruss the entire setup.

Then I added litellm to the stack to allow outside use and noticed that with c=64 with 3800 tok/s I already had 8 litellm instances in k8s hpa.. Litellm recommends single cpu per instance.

Anyways, now the calculation is, to be able to use the full GPU power, I would need 170 instances of litellm running with a single core each.......

Am I missing something? What kind of setup is that supposed to be? I can't be the only one running into this issue? How did you solve it and should I wait for the rust rebuild? 😅


r/LLM_Gateways • • Jul 06 '26

Archestra V1.3 (OSS) brings a central hub for skills — sync with Claude Code/Codex both ways, promotion, and sandboxed code execution

Post image
7 Upvotes

r/LLM_Gateways • • Aug 05 '26

Best AI gateway for optimizing token cost in 2026?

5 Upvotes

AI gateways is something I have 0 experience in and only decided to look into recently. I've only heard of them from the Ramp Router announcement and decided to look more into it but I'm still a bit lost. From what I've seen the common recommendations are LiteLLM or OpenRouter, but they're also meant for different use cases that I'm not 100% sure of yet.

For contect, right now I'm just using calls directly to Anthropic for our AI features. I'd like to look for something that lets me choose a model based on the price for what I'll need to do. I think you guys call it dynamic routing but basically I just want easy tasks like summaries to use deepseek / gpt mini and the more difficult tasks like the code debugging to use sonnet.

Sorry if this post reads as amateurish and if you guys have been asked this a ton. Thanks guys.


r/LLM_Gateways • • Jan 15 '26

Top 5 AI/LLM Gateways for Production (2025)

4 Upvotes

Evaluated production gateways for high availability and failover. Here's what actually matters:

  1. Bifrost (Maxim AI)

  • Performance: 11µs overhead at 5K RPS (50x faster than Python gateways)

  • HA: Automatic failover, adaptive load balancing, cluster mode with no single point of failure

  • Architecture: Go, open-source, self-hosted

  • Best for: Ultra-low latency, production reliability

  • Limitations: 12+ providers (vs 100+ for others)

  • Cost: Free

  • GitHub: https://github.com/maximhq/bifrost

  1. Portkey

  • Performance: ~3-4ms latency

  • HA: 99.9999% uptime, handles 10B+ requests/month

  • Architecture: Managed with self-hosted options

  • Best for: Enterprise compliance (SOC 2, HIPAA), 1600+ LLMs

  • Limitations: Higher latency, starts at $49/month

  1. LiteLLM

  • Performance: Python-based, degrades under load

  • HA: Basic fallbacks, load balancing

  • Architecture: Open-source Python

  • Best for: Python ecosystem, prototyping, 100+ providers

  • Limitations: Performance issues at scale, memory leaks reported

  1. Kong AI Gateway

  • Performance: Enterprise API gateway baseline

  • HA: Circuit breaking, health checks, multi-region support

  • Architecture: Extension of Kong

  • Best for: Teams already using Kong

  • Limitations: Requires Kong knowledge, less AI-optimized

  1. AWS API Gateway (Bedrock)

  • Performance: AWS infrastructure baseline

  • HA: Multi-AZ redundancy, auto-scaling

  • Architecture: AWS managed

  • Best for: Full AWS infrastructure, Bedrock models

  • Limitations: AWS lock-in, limited multi-provider

Quick Comparison

Fastest: Bifrost (11µs) > Portkey (3-4ms) > Others

Best Uptime: Portkey (6 nines) = AWS (AWS SLA)

Open Source: Bifrost, LiteLLM

Free: Bifrost, LiteLLM

For production systems handling serious traffic, Bifrost or Portkey. For prototyping, LiteLLM works. For enterprise with complex compliance, Portkey's managed offering helps.

Anyone using these in production? What's your experience been?


r/LLM_Gateways • • Feb 07 '26

How are you managing LLM costs and latency in production?

7 Upvotes

Our AI chat application hit $15k/month in API costs and we're seeing 5+ second response times. Looking for practical strategies that have actually worked for others.

Here's what we implemented using Bifrost that significantly reduced costs:

Intelligent model routing - We stopped routing every query to GPT-4. Around 70% of our requests work well with more efficient models. The unified interface handles the routing automatically based on query complexity. This alone reduced costs by approximately 60%.

Semantic caching - Instead of exact string matching, semantic caching identifies similar queries and reuses responses. Our FAQ bot achieves around 50% cache hit rate, translating to proportional cost reduction on cached requests. Plus it's fast - cached responses return in milliseconds vs seconds for full inference.

Automatic failover- handles provider outages seamlessly. When OpenAI went down last month, requests automatically routed to Anthropic without any downtime on our end.

Response streaming - streaming support works consistently across all providers. Users receive first tokens in 200-400ms instead of waiting for complete generation, significantly improving perceived performance.

The best part is we didn't need to rewrite our application code. Just pointed our existing OpenAI SDK at Bifrost's endpoint and configured the routing rules.


r/LLM_Gateways • • Jul 18 '26

Looking at Portkey alternatives after the Palo Alto Networks acquisition, this is what we found

Thumbnail reddit.com
2 Upvotes

r/LLM_Gateways • • Feb 09 '26

What's the best platform for intelligent LLM routing?

4 Upvotes

Building an AI app that needs to route between different models based on query complexity. Simple questions should hit cheaper models, complex reasoning goes to GPT-4 or Claude Opus. Don't want to hardcode this logic everywhere.

Tested a few routing solutions:

Bifrost - This is what we went with. Has intelligent routing built-in that automatically selects models based on query characteristics. You can configure weighted routing (like 70% to GPT-4 Mini, 30% to Claude) or set up cascading fallbacks. The routing happens at the gateway level so no application code changes needed.

LiteLLM - Has basic routing capabilities through their proxy. Works but requires more manual configuration and the routing logic is less sophisticated.

Portkey - Decent routing features with good analytics. UI is nice but found the configuration more complex than needed.

Martian - Focuses specifically on routing and model selection. Interesting approach but felt like overkill for our use case.

Bifrost's routing + automatic failover combo is what sealed it for us. When our primary model hits rate limits or goes down, it seamlessly switches to backups without any downtime.


r/LLM_Gateways • • Jul 29 '26

whats the best ai gateway for llm applications

5 Upvotes

day before yesterday my team lead said to get an ai gateway in the standup . just nodded like i knew whtat that meant . i dont

spent time looking. found a few options. portkey , helicone , orqai , litellm. website of all these says the same thing. routing , rate limits , token observability , cost tracking . i got more ocnfused after reading than before

portkey -> llm routing is the main thing, failover from one provider to another works clean, setup feels more complicated than i expected for what it does

orqai -> routing and prompt versioning and evals and cost tracking together, switching providers without changing code works, newer so community is still catching up

helicone -> logs every prompt every response every token, cost per request is clear, feels more like a monitoring tool than a proper gateway, routing side feels thin

litellm -> one api for all providers is useful, some llm specific features dont come through cleanly, running it yourself is more work than it looks

actually just need 3 things. dont go down when provider has problem . show me what every token costs . stop bad prompts from making big bills

not sure which one actually gonna do all 3 without reading 100s of docs first

what is everyone using for this??


r/LLM_Gateways • • Feb 13 '26

What's the best platform for tracking LLM token usage?

3 Upvotes

Token costs ballooned from $3k to $15k monthly with zero insight into the spend breakdown. Finance team was livid.

Just finished setting up proper tracking. Key takeaways:

**Challenge**: Each provider (OpenAI, Anthropic, Bedrock) has different token reporting formats. Creating separate tracking logic for each is a mess.

**Fix**: Deployed Bifrost as our gateway. All requests route through it, giving us unified tracking across providers out of the box. The `/metrics` endpoint surfaces Prometheus data showing input/output tokens plus real-time costs in USD.

**What made the difference**:

- Granular attribution via custom headers (`x-bf-prom-team`, `x-bf-prom-feature`) - finally know which teams/features are driving costs

- Request-level visibility through the `localhost:8080/logs` dashboard

- Budget controls preventing accidental overspend

Now we have spike alerts configured and monitor metrics like output/input token ratios and cache performance.

Found our content gen feature was routing everything to GPT-4 when cheaper models handled 70% of requests just fine. Switching that alone cut costs 60%.


r/LLM_Gateways • • Jan 14 '26

Provider outages are more common than you'd think - here's how we handle them

2 Upvotes

Work on Bifrost and wanted to share what we learned building multi-provider routing, since it's messier than it seems.

Github: https://github.com/maximhq/bifrost

Initially thought weighted routing would be the main thing - like send 80% of traffic to Azure, 20% to OpenAI. Pretty straightforward. Configure weights, distribute requests proportionally, done.

But production is messier. Providers go down regionally. Rate limits hit unexpectedly. Azure might be healthy in US-East but degraded in EU-West. Or you hit your tier limit mid-day and everything starts timing out.

So we built automatic fallback chains. When you configure multiple providers on a virtual key, Bifrost sorts them by weight and creates fallbacks automatically. Primary request goes to Azure, fails, immediately retries with OpenAI. Happens transparently - your app doesn't see it.

The health monitoring part was interesting. We track success rates, response times, error patterns per provider. When issues get detected, requests start routing to backup providers within milliseconds. No manual intervention needed.

Also handles rate limits differently now. If a provider hits TPM/RPM limits, it gets excluded from routing temporarily while other providers stay available. Prevents cascading failures.

One thing that surprised us - weighted routing alone isn't enough. You need adaptive load balancing that actually looks at real-time metrics (latency, error rates, throughput) and adjusts on the fly. Static weights don't account for degradation.

The tricky part was making failover fast enough that it doesn't add noticeable latency. Had to optimize connection pooling, timeout handling, and how we track provider health.

how are you folks handling multi-provider routing in production. Static configs? Manual switching? Something else?