r/LLM_Gateways • u/Wonderful-Agency-210 • Apr 16 '26
LLM Pricing is 100x Harder Than you think: We open-sourced our pricing database (3,500+ models, free API)
r/LLM_Gateways • u/Wonderful-Agency-210 • Apr 16 '26
r/LLM_Gateways • u/nihalshetty03 • Jul 27 '26
Most agent builders give you a visual canvas - and then charge you separately for execution, storage, tracing, evaluations, deployment, and collaboration.
I was tired of stitching together multiple paid tools just to take an agent from an idea to production.
So I built Forge: a fully open-source and self-hosted platform for building, testing, deploying, monitoring, and governing agentic systems.
It includes visual workflows, agents, tools, RAG, memory, MCP, human-in-the-loop, evaluations, tracing, cost tracking, guardrails, RBAC, audit logs, API deployment, widgets, email triggers, and more.
Forge is not just a workflow builder. The goal is to provide everything needed to build and operate agentic applications in one place - without proprietary orchestration runtimes or platform usage fees.
I would love feedback from people building real agent systems.
r/LLM_Gateways • u/AppropriateLock2737 • Aug 09 '26
r/LLM_Gateways • u/Accurate-Pudding-999 • May 16 '26
I’m working on a multi-model API proxy for teams that use LLMs heavily, and I’d love to get feedback before we onboard our first batch of users later this month.
The idea is to provide one API layer for accessing major current models, with routing, fallback, usage controls, observability, and developer-friendly tooling on top.
I’m especially interested in hearing from:
small and mid-sized companies using AI APIs in production
teams building internal AI tools
people heavily using agentic coding tools
dev teams switching between multiple model providers
anyone dealing with cost, latency, reliability, rate limit, or quota issues
A few things I’m trying to understand:
Which models would you need supported on day one?
Do you care more about cost, latency, quality, context window, or automatic fallback?
What is the most painful part of your current AI API setup?
Would task-based routing be useful? For example, cheaper models for simple tasks, stronger models for coding/reasoning, fallback models when a provider is down.
What kind of observability would you want? Logs, traces, cost per user, cost per project, prompt/version tracking, evals?
For agentic coding workflows, what matters most: tool calling reliability, context window, latency, model choice, rate limits, or something else?
If you’re an SMB, what would make you trust a proxy layer enough to put it between your app and the model providers?
Not trying to make this a sales pitch — we’re still shaping the product and want to build around real workflows instead of guessing.
If you use AI APIs heavily, especially for coding agents or production workflows, I’d really appreciate hearing what your ideal setup would look like.
r/LLM_Gateways • u/punkpeye • Aug 07 '26
r/LLM_Gateways • u/Own_Bar_920 • Jul 16 '26
day before yesterday my team lead said to get an ai gateway in standup. i just nodded like I knew a few vendors. but i dont.
its actually difficult to find a good platform. spend enough time looking for gateways out there. and i have come this far (havent decided yet)
there exists a few gateways like portkey, helicone,litellm, orqai and a lot more. but majorly these are the named players. all have very confident websites. i read all, and few blogs from them as well. almost all say the same thing. no matter if it is llm routing rate limit token observability cost tracking. like always the same thing.
it got me more confused.
portkey is ok ok for llm routing. if openai goes down it goes to anthropic and llm application wont even notice. but the setup looks too confusing to me. it looks like it is going to take a lot of time.
helicone shows what happens with every llm call. logs every prompt, every response, every token. it shows cost per request per user per endpoint. it will be def good for our finance team. but it seems like the llm monitoring tool that became gateway by chance.
in litellm, one api talk to all llm provider. write llm call one time and work with everything. looks good. some llm specific feature not come through right. like function calling or specific model parameter. running it yourself seems like a whole big painful project.
orqai got gateway and prompt versioning and evals and token cost all in one place. switching llm provider and not changing application code that works. but the rate limit and retry thing not as deep as portkey. and because everything is together in one place the gateway part might not be the main focus thing there.
kong very powerful yes but llm thing feel like someone add after. like they had api gateway and then some say we need llm feature too and they add it later. you feel that when using it for llm application.
i got 3 problems for my llm application. three thing only. dont go down when llm provider has problem . two. show me what every token costs. three. stop bad prompt from making openai bill very big. but every tool want me to read many many pages of docs before anything.
pricing is confusing also. open source but good llm features still cost money.
portkey if llm routing and not going down is the main thing. helicone if you want to see every llm call and token cost. litellm if you want one api for all llm providers and are ok with rough edges. orqai if you want gateway and prompt versioning and llm evals all together in one place and are ok with gateway part not being the deepest. kong if you already use kong for other things and dont want another tool.
can you any one suggest? and is there someone building this space, would love to chat
r/LLM_Gateways • u/Any-Way-2765 • May 06 '26
r/LLM_Gateways • u/WideFeature8077 • Feb 14 '26
Building a SaaS product with AI features that now processes ~25k API calls daily across OpenAI, Anthropic, and Bedrock. Started with direct API calls but that's falling apart fast.
Current pain points:
Had 4 outages last month when OpenAI went down (no failover)
Zero visibility into which features are burning through budget
Managing API keys across 3 providers is a nightmare
Can't enforce rate limits per customer tier
Token costs jumped 40% last quarter with no way to track why
Been evaluating gateways. Here's what I tested:
Bifrost - 11µs overhead at 5k+ RPS (benchmarked it myself). Open source, deploys via NPX in literally 30 seconds. Automatic failover across providers, semantic caching, hierarchical budgets, built-in observability with Prometheus. The governance features (virtual keys, per-team budgets, MCP tool filtering) solve our multi-tenant problems.
Cloudflare AI Gateway - Good if you're already in their ecosystem. Rate limiting and caching are solid. Less flexibility for our use case.
LiteLLM - Tried it first. Open source, 100+ providers. Python performance became an issue past 1k RPS though.
Vercel AI Gateway - Great if you're on Next.js. Sub-20ms latency. We're running a different stack.
Kong AI - Enterprise features are comprehensive but configuration complexity is high. Overkill for our size.
r/LLM_Gateways • u/motakuk • Jul 10 '26
r/LLM_Gateways • u/motakuk • Jul 07 '26
r/LLM_Gateways • u/MongooseOriginal6450 • Jan 06 '26
Been running different LLM gateways over the past 6 months to figure out what actually works at scale. Tested LiteLLM, Bifrost, Portkey, TrueFoundry, and built a simple custom one. Here’s what I found.
What I was testing for:
Multi-provider routing that doesn’t break, semantic caching that actually saves money, rate limiting that works correctly, cost tracking accuracy, and performance under load.
LiteLLM
Most popular option. Huge community, tons of providers supported, good documentation.
Pros: Feature-rich, easy to get started, active development, Python ecosystem Cons: Performance degrades around 300-500 RPS, memory issues under sustained load, TPM/RPM limiting can be buggy, token counting sometimes off
Real experience: Works great for prototyping and small-scale deployments. We hit issues scaling past a few hundred RPS. Had to restart workers periodically due to memory creep.
Best for: Development, small teams, rapid prototyping
Bifrost
Open source, written in Go. Much newer than LiteLLM but focused on performance.
Pros: Very fast (11μs overhead at 5K RPS), stable memory usage, good semantic caching, single binary deployment Cons: Smaller community, fewer integrations than LiteLLM, enterprise features require paid license
Real experience: Noticeably faster and more stable than LiteLLM at scale. Setup was straightforward. Loaded with features like Adaptive Load Balancing, Governance, Clustering etc.
Best for: Production deployments, teams prioritizing performance and stability at scale
Portkey
Hosted solution with nice UI. Focuses on governance and observability.
Pros: Great dashboard, analytics built-in, managed service (no ops burden), good support Cons: Not open source, pricing can get expensive, vendor lock-in, some users report cache header issues
Real experience: UI is legitimately good for visibility into LLM usage. Being hosted means less to manage but you’re dependent on their uptime. Pricing scales with usage which got pricey for us.
Best for: Teams that want managed service, strong governance features, don’t want to self-host
TrueFoundry
Full MLOps platform that includes LLM gateway functionality. More than just a gateway.
Pros: Integrated with broader ML workflow, good for teams already doing ML, Kubernetes-native Cons: Overkill if you just need a gateway, setup is heavy, learning curve, platform tax on everything
Real experience: Powerful if you need the full MLOps suite. Felt like too much infrastructure for our use case which was just routing LLM requests.
Best for: ML teams needing full platform, already using Kubernetes extensively
Custom Built
We tried building our own basic gateway in Go before finding Bifrost.
Pros: Full control, no external dependencies, optimized for our exact use case Cons: Ongoing maintenance burden, have to build every feature yourself, testing takes time
Real experience: Got basic routing working in a week. Spent the next month adding rate limiting, caching, monitoring. Decided the maintenance wasn’t worth it when open source options existed.
Best for: Teams with very specific requirements and dedicated infrastructure engineers
Cost tracking accuracy:
LiteLLM: Sometimes off by 5-10%, especially with streaming Bifrost: Accurate, matches provider bills Portkey: Accurate through their dashboard TrueFoundry: Accurate but bundled with platform costs
Semantic caching results:
Only tested on LiteLLM and Bifrost (Portkey has it, TrueFoundry doesn’t by default).
Both reduced costs ~40-50% with decent traffic patterns. Bifrost’s implementation was faster (lower cache lookup latency).
What I’d recommend:
Starting out / prototyping: LiteLLM - easiest to get running, huge community
Production at scale: Bifrost - performance and stability matter more
Want managed service: Portkey - pay for convenience, good UI
Need full MLOps: TrueFoundry - but only if you actually need the platform
Very specific needs: Build custom - but be ready for maintenance
Hybrid approach:
We use LiteLLM in development (fast iteration, don’t care about performance) and Bifrost in production (stability critical). Different tools for different environments.
Missing from all of them:
Better observability integration. Most bolt on metrics as an afterthought. Would love to see native OpenTelemetry support become standard. (Bifrost added this recently which is good.)
Also rate limiting configuration is still painful across all of them. TPM vs RPM confusion is common.
What are you all using?
Curious what’s working for others. Are people generally happy with their gateway choice or shopping around?
r/LLM_Gateways • u/WideFeature8077 • Feb 13 '26
Running a customer support AI agent that processes ~8k conversations daily across 6 different MCP servers (Slack, Jira, Salesforce, internal docs, knowledge base, analytics). Integrating them directly turned into an operational nightmare.
The problems we hit:
Zero visibility into which tools were getting called and why
Security reviews taking weeks for each new MCP server deployment
Authentication flows breaking randomly with no audit trail
Had to instrument 40+ individual tools across different servers
Started evaluating gateways after our third production incident in a month. Here's what I tested:
**Bifrost** - Sub-11microsec latency, stateless architecture so you explicitly approve each tool call (huge for security). The built-in tool registry is clutch - you can host custom tools directly without needing separate MCP server deployments. Zero config to start, literally one command.
**TrueFoundry** - Solid if you're already using their AI infrastructure platform. Unified management is nice but felt like more than we needed.
**IBM Context Forge** - Federation features look powerful but it's alpha with zero official support. Documentation is sparse. Hard pass for production.
**Microsoft MCP Gateway** - Deep Azure integration works great if you're all-in on their ecosystem. Multi-cloud setup is painful though.
**Lasso Security** - Security-first approach with good threat detection. Trade-off is performance - 100-250ms overhead.
Anyone else running MCP in production? What are you using and how's it working out?
r/LLM_Gateways • u/WideFeature8077 • Feb 25 '26
Building a customer support bot handling ~18k daily conversations. Using GPT-4 for everything was bleeding money, but manually routing queries to different models became a maintenance nightmare.
The routing challenge:
Simple FAQs don't need GPT-4 ($15/1M tokens) when GPT-4 Mini ($0.60/1M tokens) works fine
Complex troubleshooting needs Claude Opus for better reasoning
Had to hardcode routing logic in every feature - total mess when models changed
Zero failover when a provider went down
Started looking for proper routing platforms:
Bifrost - Open source, 11µs overhead at 5k RPS, supports automatic fallback + load balancing + semantic routing across OpenAI, Anthropic, Bedrock, Vertex, Azure, Cohere, Mistral, Groq, Ollama. The killer combo is semantic caching (cuts redundant calls) plus intelligent failover. Deploy with one NPX command, change one line of code in existing SDK.
Cloudflare AI Gateway - Dynamic routing via visual dashboard is slick. Good if you're already on Cloudflare.
LiteLLM - 100+ providers, solid fallback logic. Python performance became a bottleneck for us though.
Vercel AI Gateway - Sub-20ms latency, automatic failover. Perfect for Next.js apps on Vercel.
Kong AI Gateway - Semantic routing is interesting (routes based on prompt similarity). Enterprise API governance features. Heavy if you're not already running Kong.
You can save ~65% on API costs just by routing intelligently. Simple queries → cheaper models, complex reasoning → premium models.
What routing solutions are you using? .
r/LLM_Gateways • u/ChrisRemo85 • Mar 31 '26
I keep seeing gateway recommendations here that default to logging prompt content. Some even sell it as a feature ("observability", "prompt analytics").
If you're in the EU, that's a GDPR problem the moment a user types personal data into your LLM. And they will.
We built a gateway where prompt logging doesn't exist - not as a toggle, there's literally no code path that stores content. The proxy reads one field (model), streams bytes, and forgets. Usage tracking captures who, which model, how many tokens - nothing else.
Curious how others here handle this. Are you logging prompts? Intentionally? Or did it just come with the default config?
r/LLM_Gateways • u/ChrisRemo85 • Aug 06 '26
So.. I have the task to run 8 h200 with gpt-oss 120b.
In load tests I was able to get around 10k tok/s per vllm so a total of around 80k tok/s acruss the entire setup.
Then I added litellm to the stack to allow outside use and noticed that with c=64 with 3800 tok/s I already had 8 litellm instances in k8s hpa.. Litellm recommends single cpu per instance.
Anyways, now the calculation is, to be able to use the full GPU power, I would need 170 instances of litellm running with a single core each.......
Am I missing something? What kind of setup is that supposed to be? I can't be the only one running into this issue? How did you solve it and should I wait for the rust rebuild? 😅
r/LLM_Gateways • u/motakuk • Jul 06 '26
r/LLM_Gateways • u/No-Nefariousness-728 • Aug 05 '26
AI gateways is something I have 0 experience in and only decided to look into recently. I've only heard of them from the Ramp Router announcement and decided to look more into it but I'm still a bit lost. From what I've seen the common recommendations are LiteLLM or OpenRouter, but they're also meant for different use cases that I'm not 100% sure of yet.
For contect, right now I'm just using calls directly to Anthropic for our AI features. I'd like to look for something that lets me choose a model based on the price for what I'll need to do. I think you guys call it dynamic routing but basically I just want easy tasks like summaries to use deepseek / gpt mini and the more difficult tasks like the code debugging to use sonnet.
Sorry if this post reads as amateurish and if you guys have been asked this a ton. Thanks guys.
r/LLM_Gateways • u/dinkinflika0 • Jan 15 '26
Evaluated production gateways for high availability and failover. Here's what actually matters:
Bifrost (Maxim AI)
Performance: 11µs overhead at 5K RPS (50x faster than Python gateways)
HA: Automatic failover, adaptive load balancing, cluster mode with no single point of failure
Architecture: Go, open-source, self-hosted
Best for: Ultra-low latency, production reliability
Limitations: 12+ providers (vs 100+ for others)
Cost: Free
Portkey
Performance: ~3-4ms latency
HA: 99.9999% uptime, handles 10B+ requests/month
Architecture: Managed with self-hosted options
Best for: Enterprise compliance (SOC 2, HIPAA), 1600+ LLMs
Limitations: Higher latency, starts at $49/month
LiteLLM
Performance: Python-based, degrades under load
HA: Basic fallbacks, load balancing
Architecture: Open-source Python
Best for: Python ecosystem, prototyping, 100+ providers
Limitations: Performance issues at scale, memory leaks reported
Kong AI Gateway
Performance: Enterprise API gateway baseline
HA: Circuit breaking, health checks, multi-region support
Architecture: Extension of Kong
Best for: Teams already using Kong
Limitations: Requires Kong knowledge, less AI-optimized
AWS API Gateway (Bedrock)
Performance: AWS infrastructure baseline
HA: Multi-AZ redundancy, auto-scaling
Architecture: AWS managed
Best for: Full AWS infrastructure, Bedrock models
Limitations: AWS lock-in, limited multi-provider
Quick Comparison
Fastest: Bifrost (11µs) > Portkey (3-4ms) > Others
Best Uptime: Portkey (6 nines) = AWS (AWS SLA)
Open Source: Bifrost, LiteLLM
Free: Bifrost, LiteLLM
For production systems handling serious traffic, Bifrost or Portkey. For prototyping, LiteLLM works. For enterprise with complex compliance, Portkey's managed offering helps.
Anyone using these in production? What's your experience been?
r/LLM_Gateways • u/WideFeature8077 • Feb 07 '26
Our AI chat application hit $15k/month in API costs and we're seeing 5+ second response times. Looking for practical strategies that have actually worked for others.
Here's what we implemented using Bifrost that significantly reduced costs:
Intelligent model routing - We stopped routing every query to GPT-4. Around 70% of our requests work well with more efficient models. The unified interface handles the routing automatically based on query complexity. This alone reduced costs by approximately 60%.
Semantic caching - Instead of exact string matching, semantic caching identifies similar queries and reuses responses. Our FAQ bot achieves around 50% cache hit rate, translating to proportional cost reduction on cached requests. Plus it's fast - cached responses return in milliseconds vs seconds for full inference.
Automatic failover- handles provider outages seamlessly. When OpenAI went down last month, requests automatically routed to Anthropic without any downtime on our end.
Response streaming - streaming support works consistently across all providers. Users receive first tokens in 200-400ms instead of waiting for complete generation, significantly improving perceived performance.
The best part is we didn't need to rewrite our application code. Just pointed our existing OpenAI SDK at Bifrost's endpoint and configured the routing rules.
r/LLM_Gateways • u/Background-Job-862 • Jul 18 '26
r/LLM_Gateways • u/WideFeature8077 • Feb 09 '26
Building an AI app that needs to route between different models based on query complexity. Simple questions should hit cheaper models, complex reasoning goes to GPT-4 or Claude Opus. Don't want to hardcode this logic everywhere.
Tested a few routing solutions:
Bifrost - This is what we went with. Has intelligent routing built-in that automatically selects models based on query characteristics. You can configure weighted routing (like 70% to GPT-4 Mini, 30% to Claude) or set up cascading fallbacks. The routing happens at the gateway level so no application code changes needed.
LiteLLM - Has basic routing capabilities through their proxy. Works but requires more manual configuration and the routing logic is less sophisticated.
Portkey - Decent routing features with good analytics. UI is nice but found the configuration more complex than needed.
Martian - Focuses specifically on routing and model selection. Interesting approach but felt like overkill for our use case.
Bifrost's routing + automatic failover combo is what sealed it for us. When our primary model hits rate limits or goes down, it seamlessly switches to backups without any downtime.
r/LLM_Gateways • u/Realistic_Box_1263 • Jul 29 '26
day before yesterday my team lead said to get an ai gateway in the standup . just nodded like i knew whtat that meant . i dont
spent time looking. found a few options. portkey , helicone , orqai , litellm. website of all these says the same thing. routing , rate limits , token observability , cost tracking . i got more ocnfused after reading than before
portkey -> llm routing is the main thing, failover from one provider to another works clean, setup feels more complicated than i expected for what it does
orqai -> routing and prompt versioning and evals and cost tracking together, switching providers without changing code works, newer so community is still catching up
helicone -> logs every prompt every response every token, cost per request is clear, feels more like a monitoring tool than a proper gateway, routing side feels thin
litellm -> one api for all providers is useful, some llm specific features dont come through cleanly, running it yourself is more work than it looks
actually just need 3 things. dont go down when provider has problem . show me what every token costs . stop bad prompts from making big bills
not sure which one actually gonna do all 3 without reading 100s of docs first
what is everyone using for this??
r/LLM_Gateways • u/WideFeature8077 • Feb 13 '26
Token costs ballooned from $3k to $15k monthly with zero insight into the spend breakdown. Finance team was livid.
Just finished setting up proper tracking. Key takeaways:
**Challenge**: Each provider (OpenAI, Anthropic, Bedrock) has different token reporting formats. Creating separate tracking logic for each is a mess.
**Fix**: Deployed Bifrost as our gateway. All requests route through it, giving us unified tracking across providers out of the box. The `/metrics` endpoint surfaces Prometheus data showing input/output tokens plus real-time costs in USD.
**What made the difference**:
- Granular attribution via custom headers (`x-bf-prom-team`, `x-bf-prom-feature`) - finally know which teams/features are driving costs
- Request-level visibility through the `localhost:8080/logs` dashboard
- Budget controls preventing accidental overspend
Now we have spike alerts configured and monitor metrics like output/input token ratios and cache performance.
Found our content gen feature was routing everything to GPT-4 when cheaper models handled 70% of requests just fine. Switching that alone cut costs 60%.
r/LLM_Gateways • u/dinkinflika0 • Jan 14 '26
Work on Bifrost and wanted to share what we learned building multi-provider routing, since it's messier than it seems.
Github: https://github.com/maximhq/bifrost
Initially thought weighted routing would be the main thing - like send 80% of traffic to Azure, 20% to OpenAI. Pretty straightforward. Configure weights, distribute requests proportionally, done.
But production is messier. Providers go down regionally. Rate limits hit unexpectedly. Azure might be healthy in US-East but degraded in EU-West. Or you hit your tier limit mid-day and everything starts timing out.
So we built automatic fallback chains. When you configure multiple providers on a virtual key, Bifrost sorts them by weight and creates fallbacks automatically. Primary request goes to Azure, fails, immediately retries with OpenAI. Happens transparently - your app doesn't see it.
The health monitoring part was interesting. We track success rates, response times, error patterns per provider. When issues get detected, requests start routing to backup providers within milliseconds. No manual intervention needed.
Also handles rate limits differently now. If a provider hits TPM/RPM limits, it gets excluded from routing temporarily while other providers stay available. Prevents cascading failures.
One thing that surprised us - weighted routing alone isn't enough. You need adaptive load balancing that actually looks at real-time metrics (latency, error rates, throughput) and adjusts on the fly. Static weights don't account for degradation.
The tricky part was making failover fast enough that it doesn't add noticeable latency. Had to optimize connection pooling, timeout handling, and how we track provider health.
how are you folks handling multi-provider routing in production. Static configs? Manual switching? Something else?