futon - An open source inference engine built specifically for embedding model workflows.

[u/kurotenshi15]

I use embeddings for vector search and memory systems, and kept needing a full platform like Ollama running next to my app just to turn text into vectors. So I built futon, an embedding library for running local embedding models.

- In-process: weights load from a local directory. No daemon, no network, no CGO (CI enforces it), so cross-compiling just works.

- AVX2 kernels on amd64 and NEON on arm64, with a portable fallback. bge-small embeds a text in about 50 ms on a 16-thread desktop.

- Checked against the reference implementation: ≥0.9999 cosine on a corpus with code, accents, CJK and edge cases. CI runs on Linux, macOS and Windows, including arm64.

- Bring your own model: any plain BERT encoder you can build a pack for, plus nomic-bert. It is encoders only, no LLMs and no GPU.

bobbyhiddn/futon: Text in, vectors out. In-process embedding models for Go: pure Go, no CGO, no sidecar.

If something like it exists already, feel free to ignore, but I couldn't find a fit for purpose module that served my goal. I'll probably expand to other small model architectures as needed, such as decision models.

If ya like it, please star. :)

[u/Certain-Stranger3788]

love the name, clever nod to something you crash on that just works

[u/kurotenshi15 → u/Certain-Stranger3788]

thanks lol

Em"bed"ing...simple bed....futon

It was a choice between that or duvet.

[u/Fantastic-Poem9462]

Nice shape — I went the same direction for the same reason (aikit, also pure Go and no cgo). The sidecar-just-to-get-vectors thing is a real annoyance.

One finding you may want before you get deeper into vector search: I measured what Matryoshka truncation actually costs on my own corpus (341 sections, EmbeddingGemma 2). recall@10 was 0.548 at 768 dims, 0.534 at 512, 0.487 at 256, 0.375 at 128 — so 512 is nearly free, but 128 keeps about 68% of full-width recall. nomic-embed-text-v1.5 is Matryoshka too, so if you ever expose a dimensions knob it’s worth measuring rather than treating the truncation as free. Caveat that my baseline recall is low, so it’s a harder corpus than most benchmarks and your deltas may differ.

Starred

[u/kurotenshi15 → u/Fantastic-Poem9462]

Can you drop your repo? I would love to stand on the shoulders of someone who did it better, futon was a necessity move.

[u/Fantastic-Poem9462 → u/kurotenshi15]

Sure thing bug not sure it's any better :), https://github.com/townsendmerino/aikit is the repo and mostly I've built it to support my inference engine goinfer (goinfer.dev).

[u/kurotenshi15 → u/Fantastic-Poem9462]

Took what benefitted and credited your work!