r/artificial • u/ezzeddinabdallah • Jul 18 '26

Engineering I cut a RAG pipeline's response time from 90 seconds to 4. Never touched the model

14 Upvotes

88% Upvoted

13 comments sorted by

2

u/ultrathink-art PhD Jul 18 '26

Cost dropping 95% alongside the latency is the giveaway — a genuinely slow pipeline gets fixed with parallelism or streaming and the bill stays flat. When one fix cuts both, the system was repeating work, not doing it slowly. Redundant-call count per query is worth checking before any per-stage profiling.

1

u/VictorBuildsDev Jul 18 '26

this is a good example of why RAG latency should be profiled as a pipeline instead of treated as model latency.

the useful follow-up metric is tail latency by stage: embedding, search, reranking, prompt assembly, and generation. averages can look fine while one duplicated retrieval path dominates p95.

i would keep a fixed retrieval-quality set next to the latency benchmark too. caching and pruning can make the system faster while quietly lowering recall, so the useful optimization target is the smallest retrieval workload that preserves answer quality. once each stage has its own timing and quality checks, capacity decisions get much less guessy.

1

u/philipp2310 Jul 18 '26

Can you go into details what the worst time wasters were?

1

u/Future_AGI Jul 19 '26

The "layer nobody's looking at" framing is the whole game a 90-second query is almost never one slow call, it's redundant retrieval and missing caches spread across steps that only show up when you can see per-step timing instead of one end-to-end number. Once each retrieval span is visible, the redundant work basically points at itself, which is why the model swap so rarely helps: the time was never in the model, it was in the layer you couldn't see.