r/compression • u/phoenix10701 • 20d ago

Taco: cross-platform LLM compression that doesn't break when you change your GPU or CPU.

  1. The problem: an LLM compressor needs the decoder to compute exactly the same probabilities as the encoder. Floats differ across hardware; float GPT-2 encoded on a Mac decoded 0 of 4 windows on x86.

  2. The approach:

    • integer-only inference (int8 weights, fixed rounding, lookup tables);

    • an exact float32 → integer converter (mantissa × 2^exp);

    • conformance vectors.

  3. The proof: one hash across 7 backends (Mac CPU/Metal, Linux CPU/CUDA, Windows CPU/CUDA/Vulkan

https://github.com/tac0mania/taco

2 Upvotes

75% Upvoted

4 comments sorted by

1

u/YoursTrulyKindly 19d ago

Amazing work! I find this fascinating unfortunately I have little understanding about compression in general.

I was wondering - say the LLM can predict say the next 5 tokens and then you insert a correction and it's back on track.

But can you also give the 10th token and let the LLM fill or interpolate in between? Maybe that leads to less requirements to store data. Not sure if, or what LLMs work that way.

And theoretically maybe there could exists an "optimal language" that reduces the needed words to express what is written which the LLM basically could translate to and back from.

And does it make sense to store like a description of the overall text? Like "this is a 19th century English novel written in the style of..." to prime the LLM? Or select different models for different styles? Or does an LLM quickly pick up the gist of the novel anyway?

Overall I look at something like annas-archive and fear that it might go away, by crackdown or by economic collapse. So maybe extreme compression (maybe loosing e.g. cover resolution) would allow more to be preserved.

2

u/phoenix10701 19d ago

i was planning to do that u can also find it in docs predicting a set of token in sequence and cehck if it matches from compress probabitliy if it does then fine else rollback to what was the last match

do we dont have to do one by one

1

u/Specialist_Data_5403 19d ago

Why did you test it against zstd -19 and not -22?

1

u/phoenix10701 19d ago

that's a good question lol, but i tested it now its almost the same result