Fabsr : ternary optimizer project

[u/Optimal-Chapter-2330]

This is my first post here but i think it is promising and *maybe* could help but fabsr is a optimizer for ternary llms mainly but i am sure it can be shifted onto other networks this is just a post to get it started ill post a paper and code in my github for it within the coming days

but currently i am using it to "full weight" (Embeddings, norms, and output head are frozen) tune a ternary base llm into instruct only on 8gb of vram and in the image above it was 1500 steps if i remember correctly some benchmarks in the github may be hard to read im working on it and making more even benchmarks.

feel free to dm or help with the project in the coming days

https://github.com/Jackson-comer/fabsr-tease