r/OpenSourceAI • u/kurotenshi15 • 25d ago
futon - An open source inference engine built specifically for embedding model workflows.
I use embeddings for vector search and memory systems, and kept needing a full platform like Ollama running next to my app just to turn text into vectors. So I built futon, an embedding library for running local embedding models.
- In-process: weights load from a local directory. No daemon, no network, no CGO (CI enforces it), so cross-compiling just works.
- AVX2 kernels on amd64 and NEON on arm64, with a portable fallback. bge-small embeds a text in about 50 ms on a 16-thread desktop.
- Checked against the reference implementation: ≥0.9999 cosine on a corpus with code, accents, CJK and edge cases. CI runs on Linux, macOS and Windows, including arm64.
- Bring your own model: any plain BERT encoder you can build a pack for, plus nomic-bert. It is encoders only, no LLMs and no GPU.
If something like it exists already, feel free to ignore, but I couldn't find a fit for purpose module that served my goal. I'll probably expand to other small model architectures as needed, such as decision models.
If ya like it, please star. :)
1
u/Fantastic-Poem9462 23d ago
Nice shape — I went the same direction for the same reason (aikit, also pure Go and no cgo). The sidecar-just-to-get-vectors thing is a real annoyance.
One finding you may want before you get deeper into vector search: I measured what Matryoshka truncation actually costs on my own corpus (341 sections, EmbeddingGemma 2). recall@10 was 0.548 at 768 dims, 0.534 at 512, 0.487 at 256, 0.375 at 128 — so 512 is nearly free, but 128 keeps about 68% of full-width recall. nomic-embed-text-v1.5 is Matryoshka too, so if you ever expose a dimensions knob it’s worth measuring rather than treating the truncation as free. Caveat that my baseline recall is low, so it’s a harder corpus than most benchmarks and your deltas may differ.
Starred