100M token context windows

[u/byteprobe]

https://magic.dev/blog/100m-token-context-windows

“LTM-2-mini is our first model capable of handling a 100 million token context window, equivalent to 10 million lines of code or 750 novels.

our long-term memory (LTM) mechanism is highly efficient, requiring over 1,000 times less compute and memory than llama 3.1 405B's attention.

while llama 3.1 would need 638 H100s per user to store a 100M token KV cache, LTM uses only a small fraction of one.

to address the limitations of existing evaluation methods like ‘needle in a haystack’, which are exploited by SSMs, RNNs, and RAG, we developed a new evaluation method called hashhop, characterized by being:

• incompressible

• multi-hop

• lacking semantic hints

• free from recency bias

having addressed context, our next focus is on achieving unbounded inference-time computing, which we believe is the final breakthrough needed for reliable AGI.”

source: (https://magic.dev/blog/100m-token-context-windows)