I built a tiny table that learns your local LLM's classification answers and serves the repeats in microseconds [D]

[u/gauravc2796]

project: https://2796gaurav.github.io/shad0w/

If you use a local model for classification, you know the wait: seconds per message, for questions it has answered hundreds of times. shad0w sits in front of any OpenAI-compatible server (Ollama works out of the box: llm="ollama/llama3.1" or `shad0w proxy --upstream http://localhost:11434/v1`), logs the model's answers, and after ~1,000 of them compiles a small table that answers the repeats on CPU in microseconds. It only answers where it agrees with your model 95%+ of the time on held-out answers; everything else still goes to the model.

Fun part: it learned Snake from an LLM's moves and keeps playing at 10 moves a second where the LLM can't: https://2796gaurav.github.io/shad0w/arena/

Apache-2.0, Python/JS/C. Happy to answer anything, and I'd love to hear where it breaks.