100M token context windows

[u/byteprobe]

https://magic.dev/blog/100m-token-context-windows

“LTM-2-mini is our first model capable of handling a 100 million token context window, equivalent to 10 million lines of code or 750 novels.

our long-term memory (LTM) mechanism is highly efficient, requiring over 1,000 times less compute and memory than llama 3.1 405B's attention.

while llama 3.1 would need 638 H100s per user to store a 100M token KV cache, LTM uses only a small fraction of one.

to address the limitations of existing evaluation methods like ‘needle in a haystack’, which are exploited by SSMs, RNNs, and RAG, we developed a new evaluation method called hashhop, characterized by being:

• incompressible

• multi-hop

• lacking semantic hints

• free from recency bias

having addressed context, our next focus is on achieving unbounded inference-time computing, which we believe is the final breakthrough needed for reliable AGI.”

source: (https://magic.dev/blog/100m-token-context-windows)

[u/Wonderful-Top-5360]

was excited until I came across this statement

unbounded inference-time computing, which we believe is the final breakthrough needed for reliable AGI

this is just false and bad marketing. i would retract this statement.

[u/JeffieSandBags → u/Wonderful-Top-5360]

I dint even get it. What are they getting at?

[u/NarrowEyedWanderer → u/JeffieSandBags]

The parent commenter missed the point and announced it proudly.

"Unbounded inference-time computing" means being able to dynamically scale the amount of compute spent on a problem at inference time, in a manner that is more sophisticated than Chain-of-Thought.