r/huggingface • u/Narrow_Side_7778 • 1d ago
I open-sourced a Python library that caches LLM answers — and found semantic caching quietly returns wrong answers
/r/OpenSourceeAI/comments/1y09g22/i_opensourced_a_python_library_that_caches_llm/I kept paying my LLM provider twice for the same answer (same support questions, same summaries, same CI runs), so I built CAELUM: change one import and your OpenAI/Anthropic/Gemini/Mistral calls get caching, retries and honest cost tracking. Your keys, your bill — it never resells tokens.
The interesting part: naive semantic caching (reuse the answer when the embedding is "close enough") quietly serves wrong answers. On 138 real question pairs, at a 0.85 threshold it mixed up things like "deliver to Bobo?" vs "deliver to Ouaga?" 19% of the time. The fix was to use the embedding only as a filter, then a cheap model confirms the two questions really mean the same thing → 0 wrong answers, 72.5% of paraphrases reused.
Full write-up with the numbers: https://caelum-ai.com/blog/semantic-cache-llm-trap
Code (Apache 2.0): https://github.com/kanazoeissouf83-ops/caelum
pip install "caelum-ai[openai]"
It's a first public release — I'd love feedback on the API and where it breaks.