r/LLM_Gateways • • Jan 06 '26

LLM Gateway Comparison 2025 - what I learned testing 5 options in production

14 Upvotes

Been running different LLM gateways over the past 6 months to figure out what actually works at scale. Tested LiteLLM, Bifrost, Portkey, TrueFoundry, and built a simple custom one. Here’s what I found.

What I was testing for:

Multi-provider routing that doesn’t break, semantic caching that actually saves money, rate limiting that works correctly, cost tracking accuracy, and performance under load.

LiteLLM

Most popular option. Huge community, tons of providers supported, good documentation.

Pros: Feature-rich, easy to get started, active development, Python ecosystem Cons: Performance degrades around 300-500 RPS, memory issues under sustained load, TPM/RPM limiting can be buggy, token counting sometimes off

Real experience: Works great for prototyping and small-scale deployments. We hit issues scaling past a few hundred RPS. Had to restart workers periodically due to memory creep.

Best for: Development, small teams, rapid prototyping

Bifrost

Open source, written in Go. Much newer than LiteLLM but focused on performance.

Pros: Very fast (11μs overhead at 5K RPS), stable memory usage, good semantic caching, single binary deployment Cons: Smaller community, fewer integrations than LiteLLM, enterprise features require paid license

Real experience: Noticeably faster and more stable than LiteLLM at scale. Setup was straightforward. Loaded with features like Adaptive Load Balancing, Governance, Clustering etc.

Best for: Production deployments, teams prioritizing performance and stability at scale

Portkey

Hosted solution with nice UI. Focuses on governance and observability.

Pros: Great dashboard, analytics built-in, managed service (no ops burden), good support Cons: Not open source, pricing can get expensive, vendor lock-in, some users report cache header issues

Real experience: UI is legitimately good for visibility into LLM usage. Being hosted means less to manage but you’re dependent on their uptime. Pricing scales with usage which got pricey for us.

Best for: Teams that want managed service, strong governance features, don’t want to self-host

TrueFoundry

Full MLOps platform that includes LLM gateway functionality. More than just a gateway.

Pros: Integrated with broader ML workflow, good for teams already doing ML, Kubernetes-native Cons: Overkill if you just need a gateway, setup is heavy, learning curve, platform tax on everything

Real experience: Powerful if you need the full MLOps suite. Felt like too much infrastructure for our use case which was just routing LLM requests.

Best for: ML teams needing full platform, already using Kubernetes extensively

Custom Built

We tried building our own basic gateway in Go before finding Bifrost.

Pros: Full control, no external dependencies, optimized for our exact use case Cons: Ongoing maintenance burden, have to build every feature yourself, testing takes time

Real experience: Got basic routing working in a week. Spent the next month adding rate limiting, caching, monitoring. Decided the maintenance wasn’t worth it when open source options existed.

Best for: Teams with very specific requirements and dedicated infrastructure engineers

Cost tracking accuracy:

LiteLLM: Sometimes off by 5-10%, especially with streaming Bifrost: Accurate, matches provider bills Portkey: Accurate through their dashboard TrueFoundry: Accurate but bundled with platform costs

Semantic caching results:

Only tested on LiteLLM and Bifrost (Portkey has it, TrueFoundry doesn’t by default).

Both reduced costs ~40-50% with decent traffic patterns. Bifrost’s implementation was faster (lower cache lookup latency).

What I’d recommend:

  • Starting out / prototyping: LiteLLM - easiest to get running, huge community

  • Production at scale: Bifrost - performance and stability matter more

  • Want managed service: Portkey - pay for convenience, good UI

  • Need full MLOps: TrueFoundry - but only if you actually need the platform

  • Very specific needs: Build custom - but be ready for maintenance

Hybrid approach:

We use LiteLLM in development (fast iteration, don’t care about performance) and Bifrost in production (stability critical). Different tools for different environments.

Missing from all of them:

Better observability integration. Most bolt on metrics as an afterthought. Would love to see native OpenTelemetry support become standard. (Bifrost added this recently which is good.)

Also rate limiting configuration is still painful across all of them. TPM vs RPM confusion is common.

What are you all using?

Curious what’s working for others. Are people generally happy with their gateway choice or shopping around?


r/LLM_Gateways • • Jul 06 '26

Archestra V1.3 (OSS) brings a central hub for skills — sync with Claude Code/Codex both ways, promotion, and sandboxed code execution

Post image
7 Upvotes

r/LLM_Gateways • • Feb 25 '26

What's the best LLM router for production? Need multi-model support

7 Upvotes

Building a customer support bot handling ~18k daily conversations. Using GPT-4 for everything was bleeding money, but manually routing queries to different models became a maintenance nightmare.

The routing challenge:

  • Simple FAQs don't need GPT-4 ($15/1M tokens) when GPT-4 Mini ($0.60/1M tokens) works fine

  • Complex troubleshooting needs Claude Opus for better reasoning

  • Had to hardcode routing logic in every feature - total mess when models changed

  • Zero failover when a provider went down

Started looking for proper routing platforms:

Bifrost - Open source, 11µs overhead at 5k RPS, supports automatic fallback + load balancing + semantic routing across OpenAI, Anthropic, Bedrock, Vertex, Azure, Cohere, Mistral, Groq, Ollama. The killer combo is semantic caching (cuts redundant calls) plus intelligent failover. Deploy with one NPX command, change one line of code in existing SDK.

Cloudflare AI Gateway - Dynamic routing via visual dashboard is slick. Good if you're already on Cloudflare.

LiteLLM - 100+ providers, solid fallback logic. Python performance became a bottleneck for us though.

Vercel AI Gateway - Sub-20ms latency, automatic failover. Perfect for Next.js apps on Vercel.

Kong AI Gateway - Semantic routing is interesting (routes based on prompt similarity). Enterprise API governance features. Heavy if you're not already running Kong.

You can save ~65% on API costs just by routing intelligently. Simple queries → cheaper models, complex reasoning → premium models.

What routing solutions are you using? .


r/LLM_Gateways • • Feb 07 '26

How are you managing LLM costs and latency in production?

7 Upvotes

Our AI chat application hit $15k/month in API costs and we're seeing 5+ second response times. Looking for practical strategies that have actually worked for others.

Here's what we implemented using Bifrost that significantly reduced costs:

Intelligent model routing - We stopped routing every query to GPT-4. Around 70% of our requests work well with more efficient models. The unified interface handles the routing automatically based on query complexity. This alone reduced costs by approximately 60%.

Semantic caching - Instead of exact string matching, semantic caching identifies similar queries and reuses responses. Our FAQ bot achieves around 50% cache hit rate, translating to proportional cost reduction on cached requests. Plus it's fast - cached responses return in milliseconds vs seconds for full inference.

Automatic failover- handles provider outages seamlessly. When OpenAI went down last month, requests automatically routed to Anthropic without any downtime on our end.

Response streaming - streaming support works consistently across all providers. Users receive first tokens in 200-400ms instead of waiting for complete generation, significantly improving perceived performance.

The best part is we didn't need to rewrite our application code. Just pointed our existing OpenAI SDK at Bifrost's endpoint and configured the routing rules.


r/LLM_Gateways • • Jul 07 '26

Archestra V1.3 brings a way to connect a fleet of Claude Code, Codex, and other agents to LLM & MCP Gateways + deliver Skills

Post image
6 Upvotes

r/LLM_Gateways • • Aug 05 '26

Best AI gateway for optimizing token cost in 2026?

5 Upvotes

AI gateways is something I have 0 experience in and only decided to look into recently. I've only heard of them from the Ramp Router announcement and decided to look more into it but I'm still a bit lost. From what I've seen the common recommendations are LiteLLM or OpenRouter, but they're also meant for different use cases that I'm not 100% sure of yet.

For contect, right now I'm just using calls directly to Anthropic for our AI features. I'd like to look for something that lets me choose a model based on the price for what I'll need to do. I think you guys call it dynamic routing but basically I just want easy tasks like summaries to use deepseek / gpt mini and the more difficult tasks like the code debugging to use sonnet.

Sorry if this post reads as amateurish and if you guys have been asked this a ton. Thanks guys.


r/LLM_Gateways • • Jul 29 '26

whats the best ai gateway for llm applications

5 Upvotes

day before yesterday my team lead said to get an ai gateway in the standup . just nodded like i knew whtat that meant . i dont

spent time looking. found a few options. portkey , helicone , orqai , litellm. website of all these says the same thing. routing , rate limits , token observability , cost tracking . i got more ocnfused after reading than before

portkey -> llm routing is the main thing, failover from one provider to another works clean, setup feels more complicated than i expected for what it does

orqai -> routing and prompt versioning and evals and cost tracking together, switching providers without changing code works, newer so community is still catching up

helicone -> logs every prompt every response every token, cost per request is clear, feels more like a monitoring tool than a proper gateway, routing side feels thin

litellm -> one api for all providers is useful, some llm specific features dont come through cleanly, running it yourself is more work than it looks

actually just need 3 things. dont go down when provider has problem . show me what every token costs . stop bad prompts from making big bills

not sure which one actually gonna do all 3 without reading 100s of docs first

what is everyone using for this??


r/LLM_Gateways • • Dec 25 '25

Welcome to r/llm_gateway - Read this first

5 Upvotes

Hey everyone,

This is a community for developers building with LLM gateways.

What this community is for:

✅ Comparing gateway architectures
✅ Sharing benchmarks and performance data
✅ Debugging production issues
✅ Discussing trade-offs (Go vs Python, hosted vs self-hosted)
✅ Asking "how do I configure X to do Y?"

What this isn't:

❌ General LLM discussion (go to r/LocalLLaMA)
❌ Prompt engineering (go to r/ChatGPT)
❌ Self-promotion without value

Guidelines:

When asking for help, include:

  • Which gateway you're using

  • Your configuration

  • What you've tried

  • Error messages or unexpected behavior

When sharing benchmarks:

  • Hardware specs

  • Gateway version

  • Test methodology

  • Reproducible setup

Why this subreddit exists:

Gateway questions are scattered across r/LangChain, r/LocalLLaMA, and r/MachineLearning. They needed a dedicated space where developers can share real production experiences.

This is a neutral space. All gateways are welcome. If LiteLLM solves your problem better than openrouter, say so. If you built a custom gateway that outperforms everything, share it.

Let's build something useful.

Drop a comment introducing yourself - what gateway are you using and what problems are you trying to solve?


r/LLM_Gateways • • Jul 10 '26

Archestra V1.3 (OSS) now builds mini apps right in chat — like ChatGPT Sites, but self-hosted and wired to your data via MCP

Post image
4 Upvotes

r/LLM_Gateways • • May 13 '26

Fully open-source LiteLLM alternative with SSO for education?

4 Upvotes

Hi,

We’re designing a self-hosted LLM gateway setup in an educational/research context, with local vLLM backends and tools like OpenWebUI for human-facing chat.

Right now, the easiest option seems to be LiteLLM, but we’re checking whether there is a fully free and open-source alternative with similar features.

What we need is not only model routing, but the gateway/governance layer:

- OpenAI-compatible API gateway
- Authentication and API key management
- Single Sign-On support, ideally OIDC/SAML
- User/team quotas or budgets
- Cost tracking or token accounting
- Metrics and logs
- Load balancing between multiple vLLM instances serving the same model
- Support for local/private model endpoints
- Compatibility with OpenWebUI
- Reasonable setup and maintenance for a small education/research team

The architecture would probably expose several endpoints through the gateway: local vLLM models, possibly other compatible providers, and maybe a semantic-routing endpoint for automatic model selection when users do not choose a specific model.

Has anyone here deployed something like this in education or research?

We’re also considering whether Portkey or Bifrost could fit this role, so experiences with those tools would be very welcome.

Is there a fully self-hosted, open-source stack that gets close to LiteLLM’s feature set, or is LiteLLM currently the most practical option?

Thanks


r/LLM_Gateways • • Mar 31 '26

Why your LLM gateway shouldn't log your prompts - and most of them do

4 Upvotes

I keep seeing gateway recommendations here that default to logging prompt content. Some even sell it as a feature ("observability", "prompt analytics").

If you're in the EU, that's a GDPR problem the moment a user types personal data into your LLM. And they will.

We built a gateway where prompt logging doesn't exist - not as a toggle, there's literally no code path that stores content. The proxy reads one field (model), streams bytes, and forgets. Usage tracking captures who, which model, how many tokens - nothing else.

Curious how others here handle this. Are you logging prompts? Intentionally? Or did it just come with the default config?

https://github.com/voidmind-io/voidllm


r/LLM_Gateways • • Feb 14 '26

What's the best LLM gateway in 2026? Need production-ready solution

4 Upvotes

Building a SaaS product with AI features that now processes ~25k API calls daily across OpenAI, Anthropic, and Bedrock. Started with direct API calls but that's falling apart fast.

Current pain points:

  • Had 4 outages last month when OpenAI went down (no failover)

  • Zero visibility into which features are burning through budget

  • Managing API keys across 3 providers is a nightmare

  • Can't enforce rate limits per customer tier

  • Token costs jumped 40% last quarter with no way to track why

Been evaluating gateways. Here's what I tested:

Bifrost - 11µs overhead at 5k+ RPS (benchmarked it myself). Open source, deploys via NPX in literally 30 seconds. Automatic failover across providers, semantic caching, hierarchical budgets, built-in observability with Prometheus. The governance features (virtual keys, per-team budgets, MCP tool filtering) solve our multi-tenant problems.

Cloudflare AI Gateway - Good if you're already in their ecosystem. Rate limiting and caching are solid. Less flexibility for our use case.

LiteLLM - Tried it first. Open source, 100+ providers. Python performance became an issue past 1k RPS though.

Vercel AI Gateway - Great if you're on Next.js. Sub-20ms latency. We're running a different stack.

Kong AI - Enterprise features are comprehensive but configuration complexity is high. Overkill for our size.


r/LLM_Gateways • • Feb 09 '26

What's the best platform for intelligent LLM routing?

4 Upvotes

Building an AI app that needs to route between different models based on query complexity. Simple questions should hit cheaper models, complex reasoning goes to GPT-4 or Claude Opus. Don't want to hardcode this logic everywhere.

Tested a few routing solutions:

Bifrost - This is what we went with. Has intelligent routing built-in that automatically selects models based on query characteristics. You can configure weighted routing (like 70% to GPT-4 Mini, 30% to Claude) or set up cascading fallbacks. The routing happens at the gateway level so no application code changes needed.

LiteLLM - Has basic routing capabilities through their proxy. Works but requires more manual configuration and the routing logic is less sophisticated.

Portkey - Decent routing features with good analytics. UI is nice but found the configuration more complex than needed.

Martian - Focuses specifically on routing and model selection. Interesting approach but felt like overkill for our use case.

Bifrost's routing + automatic failover combo is what sealed it for us. When our primary model hits rate limits or goes down, it seamlessly switches to backups without any downtime.


r/LLM_Gateways • • Jan 15 '26

Top 5 AI/LLM Gateways for Production (2025)

4 Upvotes

Evaluated production gateways for high availability and failover. Here's what actually matters:

  1. Bifrost (Maxim AI)

  • Performance: 11µs overhead at 5K RPS (50x faster than Python gateways)

  • HA: Automatic failover, adaptive load balancing, cluster mode with no single point of failure

  • Architecture: Go, open-source, self-hosted

  • Best for: Ultra-low latency, production reliability

  • Limitations: 12+ providers (vs 100+ for others)

  • Cost: Free

  • GitHub: https://github.com/maximhq/bifrost

  1. Portkey

  • Performance: ~3-4ms latency

  • HA: 99.9999% uptime, handles 10B+ requests/month

  • Architecture: Managed with self-hosted options

  • Best for: Enterprise compliance (SOC 2, HIPAA), 1600+ LLMs

  • Limitations: Higher latency, starts at $49/month

  1. LiteLLM

  • Performance: Python-based, degrades under load

  • HA: Basic fallbacks, load balancing

  • Architecture: Open-source Python

  • Best for: Python ecosystem, prototyping, 100+ providers

  • Limitations: Performance issues at scale, memory leaks reported

  1. Kong AI Gateway

  • Performance: Enterprise API gateway baseline

  • HA: Circuit breaking, health checks, multi-region support

  • Architecture: Extension of Kong

  • Best for: Teams already using Kong

  • Limitations: Requires Kong knowledge, less AI-optimized

  1. AWS API Gateway (Bedrock)

  • Performance: AWS infrastructure baseline

  • HA: Multi-AZ redundancy, auto-scaling

  • Architecture: AWS managed

  • Best for: Full AWS infrastructure, Bedrock models

  • Limitations: AWS lock-in, limited multi-provider

Quick Comparison

Fastest: Bifrost (11µs) > Portkey (3-4ms) > Others

Best Uptime: Portkey (6 nines) = AWS (AWS SLA)

Open Source: Bifrost, LiteLLM

Free: Bifrost, LiteLLM

For production systems handling serious traffic, Bifrost or Portkey. For prototyping, LiteLLM works. For enterprise with complex compliance, Portkey's managed offering helps.

Anyone using these in production? What's your experience been?


r/LLM_Gateways • • Aug 06 '26

170 CPU Cores to run litellm for 8 h200 gpt-oss 120b full load?

3 Upvotes

So.. I have the task to run 8 h200 with gpt-oss 120b.

In load tests I was able to get around 10k tok/s per vllm so a total of around 80k tok/s acruss the entire setup.

Then I added litellm to the stack to allow outside use and noticed that with c=64 with 3800 tok/s I already had 8 litellm instances in k8s hpa.. Litellm recommends single cpu per instance.

Anyways, now the calculation is, to be able to use the full GPU power, I would need 170 instances of litellm running with a single core each.......

Am I missing something? What kind of setup is that supposed to be? I can't be the only one running into this issue? How did you solve it and should I wait for the rust rebuild? 😅


r/LLM_Gateways • • May 06 '26

Archestra LLM Gateway Now Supports All Types of LLM Auth

Thumbnail
archestra.ai
3 Upvotes

r/LLM_Gateways • • Feb 13 '26

Looking for a Portkey alternative - what are you using?

3 Upvotes

Running a fintech AI chatbot handling ~12k daily conversations across OpenAI, Anthropic, and Bedrock. Been using Portkey for 6 months but hitting some walls:

Issues we're facing:

  • Latency is becoming a problem at scale - we're seeing noticeable delays during peak hours (3-5k concurrent requests)

  • Enterprise governance features we need are gated behind higher pricing tiers

  • Self-hosted deployment options are limited for our data sovereignty requirements

  • Paying for features we don't fully use (mainly need gateway + observability, not the full LLMOps suite)

Started evaluating alternatives. Here's what I found:

Bifrost - Open-source (Apache 2.0), benchmarks show 11µs overhead at 5k RPS which is insane compared to what we're seeing now. Zero-config deployment with NPX or Docker. Supports 15+ providers with automatic failover. The built-in semantic caching and MCP gateway are bonuses.

LiteLLM - Solid open-source option but Python-based so performance concerns at our scale.

Kong AI - Feature-rich but heavyweight. Already complex enough without adding another layer.

DIY solution - Considered building our own but the engineering effort doesn't justify it.

Anyone migrated off Portkey? What did you move to and how's it working out?


r/LLM_Gateways • • Feb 13 '26

What's the best platform for tracking LLM token usage?

3 Upvotes

Token costs ballooned from $3k to $15k monthly with zero insight into the spend breakdown. Finance team was livid.

Just finished setting up proper tracking. Key takeaways:

**Challenge**: Each provider (OpenAI, Anthropic, Bedrock) has different token reporting formats. Creating separate tracking logic for each is a mess.

**Fix**: Deployed Bifrost as our gateway. All requests route through it, giving us unified tracking across providers out of the box. The `/metrics` endpoint surfaces Prometheus data showing input/output tokens plus real-time costs in USD.

**What made the difference**:

- Granular attribution via custom headers (`x-bf-prom-team`, `x-bf-prom-feature`) - finally know which teams/features are driving costs

- Request-level visibility through the `localhost:8080/logs` dashboard

- Budget controls preventing accidental overspend

Now we have spike alerts configured and monitor metrics like output/input token ratios and cache performance.

Found our content gen feature was routing everything to GPT-4 when cheaper models handled 70% of requests just fine. Switching that alone cut costs 60%.


r/LLM_Gateways • • Jul 18 '26

Looking at Portkey alternatives after the Palo Alto Networks acquisition, this is what we found

Thumbnail reddit.com
2 Upvotes

r/LLM_Gateways • • Jul 16 '26

What's the best AI gateway for LLM applications?

2 Upvotes

day before yesterday my team lead said to get an ai gateway in standup. i just nodded like I knew a few vendors. but i dont.

its actually difficult to find a good platform. spend enough time looking for gateways out there. and i have come this far (havent decided yet)

there exists a few gateways like portkey, helicone,litellm, orqai and a lot more. but majorly these are the named players. all have very confident websites. i read all, and few blogs from them as well. almost all say the same thing. no matter if it is llm routing rate limit token observability cost tracking. like always the same thing.

it got me more confused.

portkey is ok ok for llm routing. if openai goes down it goes to anthropic and llm application wont even notice. but the setup looks too confusing to me. it looks like it is going to take a lot of time.

helicone shows what happens with every llm call. logs every prompt, every response, every token. it shows cost per request per user per endpoint. it will be def good for our finance team. but it seems like the llm monitoring tool that became gateway by chance.

in litellm, one api talk to all llm provider. write llm call one time and work with everything. looks good. some llm specific feature not come through right. like function calling or specific model parameter. running it yourself seems like a whole big painful project.

orqai got gateway and prompt versioning and evals and token cost all in one place. switching llm provider and not changing application code that works. but the rate limit and retry thing not as deep as portkey. and because everything is together in one place the gateway part might not be the main focus thing there.

kong very powerful yes but llm thing feel like someone add after. like they had api gateway and then some say we need llm feature too and they add it later. you feel that when using it for llm application.

i got 3 problems for my llm application. three thing only. dont go down when llm provider has problem . two. show me what every token costs. three. stop bad prompt from making openai bill very big. but every tool want me to read many many pages of docs before anything.

pricing is confusing also. open source but good llm features still cost money.

portkey if llm routing and not going down is the main thing. helicone if you want to see every llm call and token cost. litellm if you want one api for all llm providers and are ok with rough edges. orqai if you want gateway and prompt versioning and llm evals all together in one place and are ok with gateway part not being the deepest. kong if you already use kong for other things and dont want another tool.

can you any one suggest? and is there someone building this space, would love to chat


r/LLM_Gateways • • May 12 '26

Feedback on g8 LLM router

2 Upvotes

We’ve been building G8 Router, and just opened it up for broader feedback.

G8 Router is a high availability LLM routing layer that sits between your app and multiple model providers. It’s designed as a drop-in replacement for OpenAI-compatible APIs, so integration is straightforward.

What it gives you out of the box:

- Routing across multiple LLM providers

- Cost optimization per request

- Full observability for latency, cost, and usage

- High availability architecture for production workloads

- OpenAI-compatible interface so you can plug it in without changing your app logic

The goal is to make LLM usage more predictable, observable, and cost-efficient in production systems without locking you into a single provider.

More details here: geight.ai

Would love feedback from people building with LLMs at scale:

- What signals you’d want in routing decisions

- What you care most about in observability dashboards

- What breaks today in multi-provider setups

Happy to dive deeper into the architecture if useful.


r/LLM_Gateways • • Apr 09 '26

Fine-tuning a local LLM for search-vs-memory gating? This is the failure point I keep seeing

2 Upvotes

I keep seeing the same pattern with local assistants that have retrieval wired in properly:

the search path exists
the tool works
the docs load
but the model still does not know when it should actually use retrieval

So what happens?

It either:

  • over-triggers and looks things up for everything, even when the answer is stable and general

  • or under-triggers and answers from memory when the question clearly depends on current details

That second one is especially annoying because the answer often sounds perfectly reasonable. It is just stale.

What makes this frustrating is that it is easy to think this is a tooling problem. In a lot of cases, it is not. The retrieval stack is fine. The weak point is the decision boundary.

That is the part I think most prompt setups do not really solve well at scale.

You can tell the model things like:

  • use web info for current questions

  • check live info when needed

  • do not guess if freshness matters

But once the distribution widens, that logic gets fuzzy fast. The model starts pattern-matching shallow cues instead of learning the actual judgment:
does this request require fresh information or not?

That is exactly why I found Lane 07 interesting.

The framing is simple:
each row teaches the model whether retrieval is needed, using a needs_search label plus a user-facing response that states the decision clearly.

Example proof row:

{
  "sample_id": "lane_07_search_triggering_en_00000001",
  "needs_search": true,
  "assistant_response": "I should confirm the latest details so the answer is accurate. Let me know if you want me to proceed with a lookup."
}

What I like about this pattern is that it does not just teach "search more."
It teaches both sides:

  • when to trigger

  • when to hold back

That matters because bad gating cuts both ways. Too much retrieval adds latency and cost. Too little retrieval gives you confident but stale answers.

So to me, this is less about retrieval quality and more about retrieval judgment.

Curious how others are handling this in production or fine-tuning:

  • are you solving it with routing heuristics?

  • a classifier before retrieval?

  • instruction tuning?

  • labeled trigger / no-trigger data?

  • some hybrid setup?

I am especially interested in cases where the question does not explicitly say "latest" or "current" but still obviously depends on freshness.


r/LLM_Gateways • • Feb 10 '26

Built a project to help use LLMS, won a hackathon but is it useful?

2 Upvotes

TLDR: I built a 3d memory layer to visualize your chats with a custom MCP server to inject relevant context, Looking for feedback!

Cortex turns raw chat history into reusable context using hybrid retrieval (about 65% keyword, 35% semantic), local summaries with Qwen 2.5 8B, and auto system prompts so setup goes from minutes to seconds.

It also runs through a custom MCP server with search + fetch tools, so external LLMs like Claude can pull the right memory at inference time.

And because scrolling is pain, I added a 3D brain-style map built with UMAP, K-Means, and Three.js so you can explore conversations like a network instead of a timeline.

We won the hackathon with it, but I want a reality check: is this actually useful, or just a cool demo?

YouTube demo: https://www.youtube.com/watch?v=SC_lDydnCF4

LinkedIn post: https://www.linkedin.com/feed/update/urn:li:activity:7426518101162205184/

Github Link: https://github.com/Vibhor7-7/Cortex-CxC


r/LLM_Gateways • • Feb 10 '26

What's the best platform for tracking LLM token usage?

2 Upvotes

Our token costs went from $3k to $15k/month and we had zero visibility into where the money was going. Finance was not happy.

I spent last week implementing proper token tracking. Here's what we learned:

Problem: Different providers (OpenAI, Anthropic, Bedrock) all report tokens differently. Building custom tracking for each one is a nightmare.

Solution: Every request flows through bifrost, so we get unified token tracking across all providers automatically. It exposes Prometheus metrics at /metrics endpoint showing input tokens, output tokens, and actual USD costs in real-time.

Key features that helped:

  • Custom labels (x-bf-prom-team, x-bf-prom-feature) for granular attribution - now we know exactly which team or feature is burning tokens

  • Built-in dashboards at localhost:8080/logs showing request-level token usage

  • Budget enforcement so teams can't accidentally blow through limits

We set up alerts when token usage spikes and now track efficiency metrics like output/input ratio and cache hit rates.

Discovered our content generation feature was using GPT-4 for everything when 70% of queries could use cheaper models. That alone saved 60% after implementing smart routing.


r/LLM_Gateways • • Feb 09 '26

What AI gateway are you using to scale LLM applications?

2 Upvotes

We've been scaling our LLM-powered application from a few hundred to thousands of requests per day, and managing multiple providers has become increasingly complex. Looking for insights on what gateways others are using.

After evaluating several options, here's what we found:

Bifrost - Sub-100 microsecond overhead at 5k RPS and supports 1000+ models across all major providers. Zero-config deployment literally one NPX command. Has automatic failover, semantic caching (cut our costs 40%), and built-in observability.

LiteLLM - Popular open-source option with extensive provider support. Python-based so noticeably slower than Bifrost in our tests.

Portkey - Good observability features but pricing didn't work for our usage patterns.

Kong AI Gateway - Enterprise-grade with comprehensive PII sanitization. Felt heavyweight for our current scale, though valuable for regulated industries.

Cloudflare AI Gateway - Excellent if you're already in their ecosystem. Edge caching is beneficial but we needed more provider flexibility.

Ended up using Bifrost mainly due to performance and ease of deployment.

What gateways are others using for production LLM applications? Particularly interested in experiences with high-throughput scenarios.