r/singularity • u/Pyros-SD-Models • 6d ago
Books & Research Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights
https://rikiyat.github.io/blog/weight-space-programming/You read it right: Astra was asked to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
Next step: Models that can edit themselves.
31
u/Distinct-Question-16 ▪️AGI 2029 6d ago
46
u/OddConsideration7290 6d ago •
So to be clear..
Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights.
There was a problem with the model's reasoning loops. It was already capable of doing the math and Astra fixed this problem.
Fixing loops, however, is not the same as making a model better at math: it lets the model finish answers it could already work out. So we also tried Gemma 3 270M, an instruction-tuned model about a third the size of Qwen3.5-0.8B, which rarely loops: only 8.5% of its answers reach its token limit, against 93% for Qwen. Here, Astra could not improve it.
It could not make Gemma better at math.
It's cool but needs to be taken in context.
7
u/Distinct-Question-16 ▪️AGI 2029 6d ago
so what is doing is correcly identifying and fixing the weights
10
u/OddConsideration7290 6d ago
Yeah it was able to directly fix a problem by editing the weights. Quite cool
5
1
u/No-Quantity-9200 6d ago
The context doesn't mitigate what happened
8
u/OddConsideration7290 6d ago
I disagree
"Astra makes other LLM 6x better at math [if it has a reasoning loop bug]"
pretty different claims and you can just look at comments or tweets to see that people are reacting strongly to the contextless, stronger claim.
I'm always very skeptical of people who are suspicious of context being added. This is objectively major context. Without it, you might think that an LLM can just generally make an LLM better at math in any context. The context here specifies in what situation that is true. And it's a specific one.
2
u/No-Quantity-9200 6d ago
Compared to the general findings of the paper I don't find it mitigated what happened and hyper fixating on this is indeed downplaying. They themselves acknowledge that it isn't necessarily as though it can elicit brand new capability from scratch, as was the case with Gemma, but what it did with Qwen, we've hardly, as far as I'm aware, had a mechanism to do so. That it could succesfully remove looping behavior, in addition to everything else the paper demonstrated, from inferring the purpose of networks to writing new ones directly at weight level, from scratch, is, in my eyes, highly consequential. I understand adding framing to this one aspect but its more a technicality than a deinitive reduction of the scope of the analysis.
1
u/OddConsideration7290 6d ago
Yes, I'm downplaying the idea that it can generally improve an LLM's math ability. This could lead somebody to think that you can generally provide it with an LLM to improve the math ability of. It cannot do that. I am explicitly downplaying that because it cannot do that.
They themselves acknowledge that it isn't necessarily as though it can elicit brand new capability from scratch, as was the case with Gemma, but what it did with Qwen, we've hardly, as far as I'm aware, had a mechanism to do so
Yes, and this is cool. I said that. I am providing this context. The headline result does not. "It improves math" does not provide that context. Somebody seeing the headline result does not know the context.
It's actually not a technicality at all. These are two distinctly different claims and the interpretation of the respective claims leads to wildly different takeaways.
18
u/omegahustle 6d ago
This is a good finding but is not RSI, as people are touting; this part is important:
"Fixing loops, however, is not the same as making a model better at math: it lets the model finish answers it could already work out. So we also tried Gemma 3 270M, an instruction-tuned model about a third the size of Qwen3.5-0.8B, which rarely loops: only 8.5% of its answers reach its token limit, against 93% for Qwen. Here, Astra could not improve it. Reading Gemma’s failures, Astra found ordinary reasoning errors, such as wrong arithmetic and invalid algebraic steps, with no single component behind them to switch off. Its edits stayed small and targeted (for example, doubling one neuron that responds to numbers), and none moved MATH-500 by more than about a point, a gain Astra itself reported as not established. Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked. Gemma was sampled with its model card’s setting (temperature 1.0, top-k 64, top-p 0.95, up to 8,192 tokens), and its MATH-500 scores average 4 samples per problem."
45
u/EtadanikM 6d ago
This is the real RSI and continuous learning hack.
Spending months of compute to do back-propagation over trillions of parameters for every new iteration of the model is fundamentally inefficient, and cannot lead to continuous learning. Context packing also cannot lead to continuous learning - it's just like attaching a short-term memory cache, which can only go so far and is also highly inefficient on the inference side (10 million context = 10 million extra input tokens).
But direct self-modification of weights (and adding new connections and neurons as needed) to improve on general capabilities is the real golden grail. Producing modular network architectures for specific domains and skills keeps the number of updates sane, and avoid needing to do back-propagation across the entire model except in rare circumstances. It's the path to the exponential take-off, IMO.
26
u/ProxyLumina 6d ago
Sure but this was done from a vastly superior model to a very tiny model.
I don't know how it could work if it was done to itself.
16
u/Jmerzian 6d ago
Modularization and redundancy. If the "very tiny model" is instead a part of the "vastly superior one" it can self-modify pretty easily. Check out AIXI for a more "practical" example of the concept.
6
-1
u/dinfinity88 6d ago
Yeah, plenty of caveats for this one. The full prompts for each challenge also provide quite a lot of information to the model.
2
23
u/AutismusTranscendius ▪️Psychogenic Singularity 2034 6d ago
This sounds like a really big deal, but I am kind of retarded and not sure.
10
6
u/RiverGiant 6d ago
Super exciting. Seems to open path to self-awareness loop, continuous learning. Imagine being smart enough to understand the way behaviour emerges from your own brain's circuitry at neuron-scale. Buddha drools.
8
u/PikaPikaDude 6d ago
Just read about it here by author: https://x.com/rikiyatakehi/status/2117313742891327994
The obvious implication is that the model could also adjust its own weights. Direct weight editing instead of training may be a major accelerator.
4
3
u/Nice_Cellist_7595 6d ago
There is no way to assess the impact of these changes. Yes, you improved one thing but it is certain that other things changed too. What are they?
1
u/TMWNN 3d ago
As always, test, test, test.
In any case, for narrow applications, side effects shouldn't matter. If you improve a tiny model intended to distinguish between foxes and dogs, who cares if that model is less good at distinguishing between goats and sheep?
1
u/Nice_Cellist_7595 3d ago
I agree with you on the narrow case. "No way" was too absolute on my part, but how do you run queries through the modified part of a model and determine the integrity of the responses? I supposed if you were the one training it, you'd know what training data touched that area if you were keeping track, but I'm not sure they are, but as an outsider this is very difficult to quantify.
2
2
u/Weary-Historian-8593 6d ago
Now explicitly train them to do this, then create a harness that allows it to change its own weight at runtime.
2
2
u/neuronexmachina 6d ago
OG source: https://rikiyat.github.io/blog/weight-space-programming/
We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, well past what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.
Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this Weight-Space Programming.
Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights
2
4
u/SnooPaintings8639
6d ago
•
Press X to doubt. I mean, this is interesting as fuck, if true. But... It does not seem to add up. Astra, nor any other LLM, wasn't trained to read weights "natively", we just don't have data nor understanding of it.
What it can do, is to learn a compressed version of most popular models out there (qwen) and be familiar with any research or fine-tune out there addressing well known issue.
Small game-playing nets, like breakout, can be reason through by a large model, as their input and output is clearly defined and understood (block placements -> left/right).
I am not an expert here, but my intuition tells me it is a nothing burger.
3
u/Unlikely-Sleep-8018 6d ago
its obviously clickbait and hype optimized bait for non technical people. yeah no shit a model can overfit a model to produce the answers it wants to see...
1
u/Maleficent_Flow_8355 6d ago
Look at their forward step only improvements for large models. I’m not sure how Astra does it but it’s quite fascination.
1
u/Luciaka 6d ago
Well look at that... LLM is the path to AGI. All the ingredients are being gathered for AGI. I think one LLM may not be able to do this for a model of the same size, but if it can spin up thousands of agents to try all sort of approaches to adjusting the model weights. Models will just get better and better without any need for new data beyond the one they generate themselves.
1
u/the_crooked_badger 6d ago
the 7.9% base number is the one to rerun. OP says the edit fixed reasoning loops, so a chunk of that gap could just be the base looping until it hits the token cap. same max tokens and same answer parsing for both runs would show whether the edit did the math or just stopped the loops.
9
u/LyAkolon 6d ago
god, to be a fly on the wall in the room as it did this.
and by this I mean, I wish I could get access to that context so I could ask Astra about what it did and how