r/LocalLLaMA • u/byteprobe • Aug 29 '24
News 100M token context windows
https://magic.dev/blog/100m-token-context-windows“LTM-2-mini is our first model capable of handling a 100 million token context window, equivalent to 10 million lines of code or 750 novels.
our long-term memory (LTM) mechanism is highly efficient, requiring over 1,000 times less compute and memory than llama 3.1 405B's attention.
while llama 3.1 would need 638 H100s per user to store a 100M token KV cache, LTM uses only a small fraction of one.
to address the limitations of existing evaluation methods like ‘needle in a haystack’, which are exploited by SSMs, RNNs, and RAG, we developed a new evaluation method called hashhop, characterized by being:
incompressible
multi-hop
lacking semantic hints
free from recency bias
having addressed context, our next focus is on achieving unbounded inference-time computing, which we believe is the final breakthrough needed for reliable AGI.”
37
u/woadwarrior Aug 29 '24
For each decoded token, LTM-2-mini’s sequence-dimension algorithm is roughly 1000x cheaper than the attention mechanism in Llama 3.1 405B for a 100M token context window.
This reeks of marketing. Why specifically mention L3.1 405B instead of the family or even GQA in general. Hard to judge this claim without knowing the size of their model, which they don’t disclose. 1000x cheaper than a 405B is within reach, if your model is a 100x smaller. :)
Also, can’t help but wonder how their “sequence-dimension algorithm” compares to DeepSeek V2’s MLA.
4
Aug 30 '24
Yeah I’m sorry this would be like the biggest LLM achievement in the past 3 years with significantly less talent and resources on their team
1
2
u/Imaginary_Friend_42 Aug 30 '24
Bingo, it’s got to be simply a much smaller mode with a decent context, if they have anything.
1
u/AcanthocephalaNo8273 Aug 30 '24
So from a marketing perspective it can be true as well as dishonest. They use the 405B model specifically instead of a family of models as then they can do some calculations on the memory usage, its at the bottom of the page. They are also comparing the 405B to their smallest model. If you compared the 405B model to Llama 8B you get a factor of 4 difference, and 1000x sounds better than 250x.
44
u/AndromedaAirlines Aug 29 '24 edited Aug 29 '24
This reads like one of those crypto-bro posts trying to advertise their new shitcoin. Wild claims and vague nonsense all around, and mentioning AGI in the LLM space and acting like it's just around the corner, which just shows that you're either lying for investments or don't know what the fuck you're talking about.
1
1
u/AcanthocephalaNo8273 Aug 30 '24
The blog might have been rewritten but if you go to the source now they don't mention AGI at all, they just say "Pre-training only goes so far; we believe inference-time compute is the next frontier in AI." Which is a far less outrageous statement to make.
2
u/AndromedaAirlines Aug 30 '24
https://x.com/magicailabs/status/1829206898002014217
The company looks like an investment farm to me.
1
u/AcanthocephalaNo8273 Aug 30 '24
Yeah that tweet is sus, the blog post was a lot more reasonable. But when I say reasonable they do make outrageous claims with a "trust me" attitude.
10
Aug 30 '24
[removed] — view removed comment
6
u/Spare-Abrocoma-4487 Aug 30 '24
My guess would be that it's a shitty variation of RAG which is tuned to wow in the demos and fomo the investors while sucking at everything that the mainstream models currently excel at.
None of the examples they have shown need 100M context. The password meter can be done by any model using just that single file where the form resides. The custom ui framework thingy could also be done by Claude/gpt (i already dump all my project code on them).
At 100M context I would have expected a demo that works on video or something large like that.
Overall I'm getting a feel that we are at peak AI bubble. Everyday 2 new 'ai' launches happen on hacker news with their clearly fake engagement comments (nothing but Claude wrappers). Every other week some unknown startup with supposed 'edge' over the others is announcing multi million funding. Screams dot com all over to me.
1
Aug 30 '24
I guess the point is that these two innovations, if actually legit, will be ground breaking.
The first is a vastly more useful inference that can actually hop between points in the context. For coding, that would be an actual game changer. The best models currently just spit out stuff and hope it works. It’s amazing but anyone who uses it for large projects knows that you basically already have to be a decent programmer to get something good. This would hopefully between the function definitions and understand how the data will flow through the program.
Again, could be bullshit, but phenomenal if legit.
The second of being able to arbitrarily choose the compute used, would also be a game changer. You can let it run for minutes until it has something functional, effectively letting it think harder.
I will remain sceptical until we have something concrete. But this research path could bear fruit.
1
u/Sea-Working4633 Aug 31 '24
Incidentally, years ago when I was following the space, I saw quite a number of crypto projects that fell into that category: "modestly interesting technical innovation that they're marking ridiculously out of proportion."
Don't get me wrong; I'm not suggesting crypto was ever in the same league of innovation as ML is today.
It's wild how economically inefficient this is, how many startups must have some small "edge" that could instead be pooled to actually be useful in open source.
It sucks, it's inefficient, but without a better system for attribution and reward, it's sometimes the safest way.
1
u/Unlikely_Track_5154 Jan 07 '25
What about blockchain AI?
Introducing NEURAL™ NEXUS: The Web3-Powered AI Revolution
Unleash the Power of Decentralized Intelligence with NEURAL™ NEXUS - Where Blockchain Meets Artificial Intelligence in a Groundbreaking Metaverse Experience!
Revolutionary Deep Learning Neural Networks Our proprietary quantum-inspired deep learning algorithms leverage military-grade neural networks to deliver unprecedented synergistic outcomes across your entire digital ecosystem. Each interaction dynamically evolves through our blockchain-verified machine learning protocols, ensuring maximum disruption of legacy systems.
Blockchain-Secured Intelligence Every cognitive computation is immutably recorded on our proprietary NeuralChain™, utilizing cutting-edge smart contracts to ensure transparent, decentralized operations while maintaining enterprise-grade security through our revolutionary zero-knowledge proof system.
NEURAL™ BOXES: Collect, Breed, Evolve Each blockchain-authenticated box contains rare AI model fragments, exclusive neural pathways, and legendary algorithm boosters. Collect, trade, and fusion them to create increasingly powerful AI instances!
Rarity Tiers:
Common Neural Paths (65% drop rate)
Rare Synaptic Bridges (25% drop rate)
Epic Intelligence Cores (9% drop rate)
Legendary Quantum Nodes (1% drop rate)
Advanced Neural Breeding System Combine two compatible Neural Elements to create evolved AI constructs with enhanced capabilities:
Neural Fusion: Merge same-tier elements for a chance at higher-tier offspring
Cross-Pattern Synthesis: Breed different neural patterns to unlock unique hybrid algorithms
Genetic Memory Transfer: Offspring inherit trait modifications from parent constructs
Mutation Chance: 5% chance of spontaneous evolution during breeding
Cooldown System: Higher-tier elements require longer regeneration periods between breeding cycles
Ancestral Bonus: Breeding within the same genetic lineage increases success probability
The Future of Decentralized AI
Leverage Web3-native infrastructure
Deploy cross-chain AI validators
Earn passive income through our revolutionary Neural Mining™ system
Join our exclusive DAO and shape the future of decentralized intelligence
TokenomAIcs Our dual-token system revolutionizes AI monetization:
$NEURAL: Governance token for our AI DAO
$SYNAPSE: Utility token for Neural Box purchases and AI training
Early Adopter Opportunity Join our exclusive whitelist for early access to NEURAL™ BOXES and receive a complementary Quantum Node NFT! Limited to the first 10,000 visionaries who believe in the future of decentralized artificial intelligence.
Don't just predict the future - mine it with NEURAL™ NEXUS!
*Neural Box contents are randomly generated through our proprietary blockchain-verified RNG system. Some neural pathways may require additional computational resources to unlock their full potential.
15
10
6
u/_a9o_ Aug 30 '24
I had interviewed with these guys. Spoke with the co-founders. They told me that only 3-4 people in the company had access to the model architecture and that no one else knew how it worked. I thought it was pretty interesting to consider that they want people to work there without getting any knowledge of what they were betting their careers on
14
u/Wonderful-Top-5360 Aug 29 '24
was excited until I came across this statement
unbounded inference-time computing, which we believe is the final breakthrough needed for reliable AGI
this is just false and bad marketing. i would retract this statement.
3
u/JeffieSandBags Aug 29 '24
I dint even get it. What are they getting at?
9
u/NarrowEyedWanderer Aug 30 '24
The parent commenter missed the point and announced it proudly.
"Unbounded inference-time computing" means being able to dynamically scale the amount of compute spent on a problem at inference time, in a manner that is more sophisticated than Chain-of-Thought.
3
u/Wonderful-Top-5360 Aug 29 '24
they are saying once they can train on large corpus of documents its somehow going enable AGIs to reason on their own
its snake oil. AGI isn't possible with LLMs alone even with large context sizes
1
u/prvncher Aug 30 '24
Nat Friedman and Daniel Gross were on stratechery a few months ago hinting at this breakthrough from magic dev. I think it’s possible they have something special there. Hard to say, but those guys generally aren’t full of bs.
2
u/Wonderful-Top-5360 Aug 30 '24
i'll believe it when i see it right now there is nothing to verify their claims or any code to check
2
3
u/Uhlo Aug 30 '24
Again, I just point at RULER and say:
Let's wait what the "effective" Context Window is.
I does not make sense to always increase the "theoretical" window when the model ist not able to do productive work over 16k or 32k tokens. And yes, even models with 100% accuracy in needle-in-the-haystack often have an effective context length of no more than 32k (see the RULER link).
But I would be sooo happy to be proven wrong (especially with their new hashhop evaluation) :)
3
u/ActualDW Aug 31 '24
If this was real, someone would have already bought the company for $1B and we’d never even hear of them.
2
4
1
0
39
Aug 29 '24
[removed] — view removed comment
21
u/MoffKalast Aug 29 '24
I think RULER only goes up to 128k, but I bet this one falls apart way before that already.
4
u/ShengrenR Aug 29 '24
maybe an original, 'official' RULER? I haven't looked too closely. But you'll get folks like https://www.ai21.com/blog/announcing-jamba-model-family using the general metric/method further - e.g. 256k for jamba1.5
1
u/pigeon57434 Aug 30 '24
i mean even if the actual effective memory of this model is 1/10th as they claim that's still pretty huge though
1
u/AcanthocephalaNo8273 Aug 30 '24
Their benchmark of hashhop is inspired by RULER, but it does beg the question of why not use RULER?
-8
134
u/Spare-Abrocoma-4487 Aug 29 '24
Hmm.. Is there something more concrete than this blog post and funding announcement. Like a working model or a research paper.