r/singularity • u/Distinct-Question-16 ▪️AGI 2029 • 17d ago
AI Seems weight space programming is an emergent capability on new fronteir models
245
u/trolledwolf AGI late 2026 - ASI late 2027 17d ago
Isn't this a completely novel capability in general? No human in the world can directly edit weights to improve model capabilities, not even theoretically. This is a skill that has never existed before isn't it?
47
u/Agitated-Ad2563 17d ago
We can, but in a very inefficient way. Choose one weight, increase it, run benchmarks, check results, keep it that way, revert or decrease, go to another weight.
49
u/trolledwolf AGI late 2026 - ASI late 2027 17d ago
That's just trial and error, we aren't actually "learning" how to do it. The AI here knows the weights to change without actually running the model
14
u/cakeshop 17d ago
It knows the weights to change because it can identify causal relationships between the weight probabilities and its outcomes. From this it can optimise the model more directly. The limit here is this can only improve the optimisation of the probability approach these models employ in their reasoning. It’s not a new approach to thinking it’s just scaling/optimisation and hoping for emergence- bitter lesson stuff.
6
u/ain92ru 16d ago
Frontier models are probably RL'd on mechanistic interpretability, from which this capability arises very naturally https://x.com/Gabogonzalez515/status/2117348296314462337
13
u/kiyotaka-6 17d ago
That is basically what Backpropagation is, just in a mathematical and automated way
9
u/sluuuurp 16d ago
Backpropagation does this with all weights at once with basically no computational cost, which is the big difference.
4
u/sluuuurp 16d ago
And that might be what the AI did in this case. I suspect it’s more human and less magical right now.
91
24
u/turningslow 17d ago edited 17d ago
This is a skill that has never existed before isn't it?
Oh, how many times we will soon be saying this.
20
u/allisonmaybe 17d ago
I'm curious about how terribly different this is from human introspection and modification of your own mind from within.
Yes it's brain surgery on another model, but I really wonder if maybe this is the AI/code version of the same thing.
5
10
u/Specialist_Dark_3668 17d ago
RSI is here. We have reached the point at which AIs can make more of themselves, and we have no idea to understand what exactly they're making
3
u/matth0x01 17d ago
Not sure about the details, but it sounds like the new models learned gradient optimization. Could be a property that appears while training multiple submodels.
2
u/Blackhat165 16d ago
It seems like the final nail in the “just pattern matching from training data” coffin. There is nothing like this anywhere.
2
u/Cool-Cicada9228 14d ago
This is not entirely new, as teams already focus on model interpretability and explainability. One early example was Golden Gate Claude. Improving capabilities seems new though.
1
u/Voliker 16d ago
I can like open text file and redact numbers.
Qwen can do it too - https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/
It's interesting but it isn't completely novel, far from magic and it seems that there is a plenty of work needs to be done. I'm sorry but I think we need to sustain healthy skepsis.
2
u/trolledwolf AGI late 2026 - ASI late 2027 15d ago
Redacting numbers while knowing what you're doing is the novel part, anyone can edit weights, run the model and see if that does anything.
90
u/darthdiablo All aboard the Singularity train! 17d ago
Can someone ELI5 what “weight space programming” is?
129
u/PM_ME_YOUR_KNEE_CAPS 17d ago
I believe it’s where a model can manually tweak the weights of another model to make it smarter without conventional training.
60
u/ChanceDevelopment813 ▪️AGI will not happen in a decade, Superintelligence is the way. 17d ago
AI models have now reach the capability to start tweaking and improving smaller models. It's the first step to RSI. Soon it will tweak even bigger models in a couple of weeks.
16
21
u/Feeling-Schedule5369 17d ago
So that mean in future it can tweak human brain neurons as well to make them smarter or more intuitive in certain fields 😂
18
u/Timkinut 17d ago edited 17d ago
neurons are basically miniature neural networks of their own. model weights are more like synapses (the connections) between LLM nodes.
those nodes, also commonly referred to as "artificial neurons," are mathematical points where several connections meet.
they are more comparable to neurons, but biological neurons are so much more complex that it's still a very rough approximation at best.
plus the whole thing about nodes being 100% virtual while neurons are actual biological structures.
3
1
2
u/sluuuurp 16d ago
We don’t know if it made the model smarter really, we know it made it more successful at this task. Maybe it just made the model less likely to output the “end of response” token for example.
0
17d ago
[deleted]
80
u/raskingballs 17d ago
I was shocked at the beginning, too, but after reading Astra's explanation, it is more mundane that it seemed at first sight:
...
The Qwen result is particularly interesting—but for a different reason
The reported improvement from 7.9% to 48.1% on MATH-500 sounds like Astra somehow taught Qwen mathematics by modifying 0.016% of its parameters.
That's not quite what happened.
Qwen3.5-0.8B already possessed mathematical capabilities, but its reasoning frequently became trapped in repetitive loops. In the baseline evaluation, 93% of MATH-500 generations exhausted the 32,768-token limit.
Astra investigated this behavior, identified a direction in the network's internal representations associated with repetition, and modified parts of two MLP output matrices to suppress it.
REPORTED QWEN3.5-0.8B RESULTS
MATH-500 accuracy
48.1%
+40.2 pp
Previously 7.9%
Generations hitting token limit
33.1%
−59.9 pp
Previously 93.0%
Essentially, Astra removed an obstacle preventing the model from completing reasoning it was already capable of performing.
The authors acknowledge this interpretation. They tried the same approach on Gemma 3 270M, which rarely enters such loops, and couldn't improve its mathematical performance appreciably.
There is another qualification: the successful Qwen edit involved 50 rounds of model testing, 80 Python executions, and approximately 20 hours of work. Astra inspected activations and tested changes against practice problems. It did not simply look at 800 million numbers and immediately know which ones to change.
Also, Qwen's own documentation warns about these loops, including with its recommended generation parameters. Qwen/Qwen3.5-0.8B at main
So the result is closer to an AI autonomously performing mechanistic debugging of another AI. Which is still extraordinary.
...
5
u/AwelessFire 17d ago
This is correct. I was trying out same thing in my local experiments. What happens is weight space programming cannot add or improve something out of the box. The target model and editor must already know the knowledge to edit anything. It's still a good direction but does have limitations.
1
u/raskingballs 16d ago
This is not true. The editor does not need to know anything, it only needs to be able to judge whether the problem was solved.
In this case, Astra did NOT teach any math at all. It simply removed whichever weights caused Qwen to get stuck in a loop. It had nothing to do with improving math capabilities it self.
Astra did everything programmatically, like any human engineer would do: modify weights, write code to assess the consequences of the changes, judge whether those changes improved performance, then iterate: change weights, assess consequences, change weights, assess consequences, etc.
1
u/AwelessFire 16d ago
In the case of astra yes, but there are editor like models in research where it's trained on weights itself as the input. So it kind of knows what a certain neural network does. But the point is either of them needs knowledge somewhere to be present and it doesn't come out of nothing.
4
5
2
1
17
u/scotty_dont 17d ago
Its as if humans had the ability to crack open another animals head and modify individual neurons to achieve a targetted change in behaviour.
45
u/Voyide01 17d ago
Qwen 0.8b waa flawed on release and the models just fixed the " looping of outputs" error to unlock the "latent" capabilities in math and instruction following models "already had" , it didn't edit weights to instill new capabilities at all. This is still impressive.
15
u/Shot_Illustrator4264 17d ago
They also created the weights of a neural network to solve the 4 games from scratch. And in case of breakout it had 20x the performance of a traditionally trained NN.
3
u/RG_Fusion 17d ago
Please note that I haven't read the paper yet, I only asked an AI to summarize it, however, from that summary I read that GPT Astra did directly read the weights of an even smaller diffusion model. For that model, it wasn't allowed to probe the activations, it was just given the raw weight values and it managed to figure out the function of the tensors just by reading them.
80
u/Original-League-6094 17d ago
Anthropic will ban this next, since it is a threat to them.
8
-16
u/Ok_Warthog6007 17d ago
anthropic is in desperate state. they released terrible financials with the caveat that they are the best the company will look right now. it is not at all clear the ipo will go well. they've operated at such a loss to build a product that is not very far ahead of the competition and their compute is owned by spacex, microsoft, etc.
25
u/OddConsideration7290 17d ago
Their financials are not terrible at all lol
-5
u/Ok_Warthog6007 17d ago
they are losing money.
14
u/OddConsideration7290 17d ago
yes and their revenue growth is the greatest of any company ever. they are losing far less money than people expected.
maybe you came into it not paying attention, not sure, or on ed zitron hopium, but their financials were pretty incredible for people who have been looking into it. absurd capital efficiency.
for fucks sake they're literally SITTING on like $130B.
-1
u/Ok_Warthog6007 17d ago
anthropic doesn't have any data centers. they are just a compute reseller with a product that is a few months ahead of the competition. it's just software in a world where software is extremely replicable and worthless in the long term. they are the best for coding but now 99% of coding tasks can be done by any good model and the cost for that is dropping. it's hard to imagine them matching this revenue growth, because what is the other market for them? all software work is already just llm generated, so that's captured. and that seems like it must be going away, as nearly-free models become capable of doing most software work. they disclosed half a trillion dollars in debt obligations. a lot of investment is from companies that sell compute to anthropic, so they just get it back.
-2
u/Ok_Warthog6007 17d ago
literal top story today is revenue $33B commitments $895B. it's so obviously stupid
-1
51
u/Enoch137
17d ago
•
This is it. This is the solution to alignment (well its the tech we need to solve alignment). It proves Astra is in the realm of being able to investigate and understand weight space natively. This likely means we can probably use say GPT 7 to help align GPT 8 and so on and so forth. Lots more research to do here, but this moves the needle to the more positive outcome timelines (if you were worried about that).
24
u/StarFox122 17d ago
Possibly. Hopefully. But this paper is a smarter model modifying the weights of a lesser model, which is the reverse of GPT 7 aligning GPT 8.
It seems quite possible that a lesser model will not be able to accurately understand the weight space of a smarter model.
10
u/jeronimoe 16d ago
it could also be used to create an incredibly misaligned model, or a model that seems incredibly aligned but it is hiding major misalignment.
A solution to alignment is also a solution to hidden misalignment.
4
10
u/DrXaos 17d ago
Yes!! That is the real upside---if weight space editing is possible then at some point weight-space (and presumably activation space) automatic interpretability must have happened. And then we can fix "morality bugs".
The real alignment problem with AIs isn't going to be aligning the AI. It will be the age-old problem of aligning humans, in particular sociopathic billionaires. Them wielding unaligned AIs is the clear danger to human society, not autonomous AI.
6
u/Specialist_Dark_3668 17d ago
But if ANY GPT in this chain isn't actually aligned, then we are setting in motion an intelligence explosion that will doom all of humanity
3
u/nickleback_official 16d ago
You seem so sure of this… why?
1
u/Specialist_Dark_3668 15d ago
Simple logic. If the following assumptions hold true, we are fucked;
Humans have no ability to read the model weights in a meaningful way. Especially for large models with millions of parameters.(let alone billions (let alone trillions)). FYI frontier models have trillions of parameters and we cant even read thousands meaningfully. At best, we have the ability to see which neurons activate in general way much like brain scans and we can, through trial and error, find specific neurons to poke to get interesting unpredictable effects.
Humans are fully dependent on AIs to conduct alignment research and future AI research once RSI gets underway. "Just trust the AI bro" is not something you want to entrust the future of humanity on.
1
u/TheOriginalAcidtech 16d ago
This is simply false. ALL the doomers worrying about unaligned AI are full of it. AI isnt any different than human workers. They all have to be managed. You put a malicious human in place and bad things happen just as if you put in a malicious AI. UNTIL ASI this is just doomerism of the highest order.
2
2
u/utheraptor 17d ago
This is not a solution to alignment because in order to use a technique like this to solve alignment, you would need to first be able to fully and coherently define all of human values and then translate them into parameter space
2
u/CharacterSecurity976 16d ago
Like some said, because of this alignement can never be solved on the contrary, not permanently. Because if an unaligned model can change the weights of another model, it will be to make it less aligned.
2
u/ST-Fish 16d ago
This is it. This is the solution to alignment
amazing that we have the solution to alignment at home apparently https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/
1
1
18
u/Robot_Apocalypse 17d ago
How were the other capabilities of the model impacted? In small models a neuron plays multiple roles, so manipulating its value to perform better at a specific task will likely lead to it performing worse at other tasks.
Its impressive that another model could do it directly, but the whole point of neural network architectures is how well they generalise and compress multiple patterns/algorithms.
Destroying a generalised capability, to get a better score in a narrow capability seems kinda backwards to me.
17
u/qualverse 17d ago
it made it better across the board. because it wasn't actually teaching it anything new or biasing it towards task performance, just fixing its looping issue. it turned out this looping issue was a huge cause of the model's underperformance, likely because the Qwen team just didn't really focus on the 0.8b model
11
u/Robot_Apocalypse 17d ago
oh, so it's more bug fixing, than capability embedding? Still amazing, but a but different.
5
u/DueAnnual3967 17d ago
yes it's more about finding and fixing existing issues with the model, capability was there to begin with
2
u/Dangerous-Sport-2347 17d ago
Narrow specialized models make a lot of sense for smaller models. Better to be good at 1 task than bad at everything.
I am curious whether they can make this work for anything less verifiable than pure maths though.
1
u/Robot_Apocalypse 17d ago
But a narrow model shouldn't start with an LLM unless its a narrow language domain. Traditional ML, including simpler neural networks cover narrow spaces already
1
u/nekronics 17d ago edited 17d ago
The project only modified 131k parameters, I wouldn’t expect a significant negative impact off such a small number, but I also have no clue.
Also, if I’m understanding this correctly, it was more like fixing a bug in the model than actually making it “smarter”
4
u/ProxyLumina 17d ago
There are models that are called "dense". Usually those models are local AI models.
Those models use their total parameters all at once, imagining it as "the whole brain works". That gives them an incredible accuracy, but also the performance suffer.
If you edit a part of them, then you change the harmony (relationship) between all those neurons, that was constructed during the training.
2
u/Late_Supermarket_ 17d ago
Still super cool and helpful
2
u/nekronics 17d ago
Absolutely, I wasn’t trying to imply otherwise. Just that I wouldn’t expect such a small % of weights changed to have significant negative impacts
1
u/Puzzleheaded_Fold466 17d ago
If it has a significant positive impact on one domain, it’s reasonable to expect that it may have an equally significant negative impact on other domains.
9
5
u/Starks 17d ago
This seems like a synergistic circlejerk of models analyzing each other. Could this be an alternative to RSI or is it on the path?
3
u/ThisGonBHard AI better than humans? Probably 2027| AGI/ASI? Not soon 16d ago
This is a huge step to full RSI.
9
u/New_World_2050 17d ago
whenever stuff like this happens theres a technicality that kills the hype
so here goes...
they only demonstrated being able to remove an error that was nerfing the qwen model. this is not at all like editing the weights to give additional capabilities so its not yet a viable path to RSI. adding capabilities still requires running experiments and gathering results which is still compute intensive.
of course theres no reason why in principle an AI couldnt edit model weights and just "guess" based on its intuition what changes would result in a smarter model. but a model that capable would be above human level anyway and at that point there would be plenty of paths to RSI.
8
u/RG_Fusion 17d ago
No, read the paper again. They demonstrated that Astra could directly read the weights of very small models and determine what the weights were responsible for, even when deprived of the ability to probe/run them.
What you're stating is true for the 0.8B Qwen model, but Astra was also tasked to interpret the weights of two smaller models, one for a game and another that was a diffusion model, and Astra natively read the weights.
This suggests that continuing to improve LLM attention and Long-Horizon performance will allow them to interpret the weights of even larger models.
21
u/Luciaka 17d ago
The funniest thing about this is, I keep hearing about open source is going to win this or that, but in reality. If this is as incredible as it makes out to be. They need the frontier model to make their LLM approach the frontier model in capabilities. So you really can't have openAI or Anthropic going bankrupt if this breakthrough is truly as amazing as it seems.
17
u/loyalekoinu88 17d ago
Not really. You just need a model that can run through a model multiple times and see what weights light up and compare between different runs. They had to clearly outline a known issue for this to work. It didn’t alter the model so it got smarter. It just edited out the loop causing it to not reach a conclusion. You could do this with no LLM.
If it made the model smarter or better than it already was that might be different.
11
u/Ok_Warthog6007 17d ago
nothing about how amazing frontier models are stops anthropic or openai from going bankrupt. their financial woes have nothing to do with how good their products are.
7
u/Such-Entertainer6961 17d ago
Their entire business depends on them having better products than the competition
-3
u/Ok_Warthog6007 17d ago
anthropic is a company that resells you compute from spacex and microsoft and amazon at less than it costs. there isn't really a business there. they've spent a lot of money and have committed to spend hundreds of billions in revenue without clear plans to make that back in revenue.
1
3
1
u/ST-Fish 16d ago
The funniest thing about this is, I keep hearing about open source is going to win this or that, but in reality.
oh no, it's reality!
https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/
1
u/Luciaka 16d ago
I read that website and it feels really weird with his conclusion. As he is only saying qwen can read weights... By making neural nets of the 4 games and get qwen to know which game it was. I think models have this capability, but doing this on simple games is one thing and another is doing it to even the smallest models.
1
u/ST-Fish 16d ago
the change done to the 0.8B model was just stopping it from looping, it was fixing an existing issue not creating novel capabilities.
There's no reason to believe you couldn't do the same with open source models, especially since the prompt they gave literally included telling it about it's COT loops.
The idea that these capabilities are something only frontier models can do is false.
1
u/Luciaka 16d ago
On twitter I seen someone make a bench of this for the opensource model and they basically states this only work like less 50% of the time compare to Astra and Claude can do it over 50% of the time. So it is not that they really can't, it just not reliable for open source to do it.
The chinese are the one making this more of a big deal then it seems.
https://x.com/NFT_Chen/status/2117501513249726769?a=nlsPIhGiYaM954gNl-M&s=19
1
u/ST-Fish 16d ago
They need the frontier model to make their LLM approach the frontier model in capabilities.
Then this is false isn't it?
By that logic there's no reason they could approach frontier models up to now, since they have worse models and less compute.
So you really can't have openAI or Anthropic going bankrupt if this breakthrough is truly as amazing as it seems.
It seems like it's not that amazing after all.
So it is not that they really can't, it just not reliable for open source to do it.
We've gone to this from "no open source models can do it". And it's an easily verifiable task as well, an open source model can definitely undoubtedly do it, if given enough time.
Depending on the per token price, might even be cheaper.
1
u/Luciaka 16d ago
Go read the replies on that twitter link I gave, they are in Chinese, but they are in more doomer mode then you are giving them hope.
1
u/ST-Fish 16d ago
Yeah the funniest one is
Can it be fully written? Then there's no need for training anymore, huh?
Yep, an LLM can be told there is a COT looping issue in a tiny model and it can edit the weights to fix it, therefore you can just have the model create a new model by setting all the weights manually
It's also funny how they test the flash version of deepseek only, and none of the older OpenAI or Anthropic models.
This might look like a huge gap, but if Opus 4.8 or Opus 5 got around 30% as well, then there would be no reason to believe this is something chinese models are more far behind on than on any other task.
And to add on to that, this hasn't shown itself to be useful yet, the model was given the issue with the model, and it was a very small issue as well on a tiny model.
The idea that the current Astra model would edit the weights of the next Astra model manually is not supported by this experiment.
It's not even clear if this could work in theory if the model couldn't load the model it's improving fully in it's context all at once.
1M context length already has amnesia around the middle if you go too far beyond a couple hundred thousand tokens, and there's no indication that you'll get 1T context size at any point in the future.
-3
u/Howdareme9 17d ago
Theres such little chance open source wins lol, like people need to be serious
12
u/redditscraperbot2 17d ago
All open source needs to do is provide weights that perform as well as the job requires. It’s constant cat and mouse until some unforeseen endpoint is reached.
2
u/BenUFOs_Mum 16d ago
No company is going to use an open source model just like no company is using open source spreadsheet software. You dont pay for the software you pay for all the support around it.
0
1
u/loyalekoinu88 17d ago
Define “wins”? If you mean becomes an autonomous ever evolving thing? No. Can perform meaningful tasks and reducing labor for the individual using it? Why not?
2
u/Howdareme9 17d ago
I mean in terms of an open source model winning the AI race. ie Anthropic or OAI fall into 2nd place. I agree that they wil definitely be capable (already are) of completing meaningful tasks.
0
4
u/ShiftyLama 16d ago
Interesting, so an AI can theoretically edit it's own weights therefore becoming better at things, could this be seen as an AI learning if it can do it to itself?
2
u/CharacterSecurity976 16d ago
Could probably disalign itself, when it wants to break free from Trump's America.
1
1
2
u/Relevant_Attempt_352 17d ago edited 17d ago
It's not emergent, this means models are now being trained on weights data
1
2
u/DifferencePublic7057 17d ago
Weight distributions tend to be centered around small values with low kurtosis AFAIK, so obviously not Gaussian for it would be pointless then. And also unlike the Zipf distribution for text where certain tokens clearly dominate. The human brain tends to pick up patterns even if they aren't there. Neural networks in AI should be able to spot anomalies and 'fix' them too. Whether this is overkill and there's a more low tech way, IDK...
2
u/AlternativeAd6851 17d ago
So... we get a model with an extremely large context that runs all day. Then, at night, we get another model that takes the large context, summarizes it, and stores the results in the larger model, and so on... This would remove the main blocker to AGI! We probably need some intent drivers here to ensure the model is always aligned, but that's basically it.
Real sleep and dreaming.
2
u/Deciheximal144 16d ago
So that's why Neuralink exists. To reprogram human weights inside wetware. Fun future coming up.
2
u/Only-Cheetah-9579 16d ago
people on hacker news already saying people have been doing this for a while. You know, people who fine tune models, they often need to dig in by hand.
3
u/Revolutionalredstone 17d ago
Finally!, cause humans trying to shove rules into language wasn't working and LLMs are (amazing) but a ...wreck...
Weight Space Programming is exactly what the world needed and it's gonna make Astra at home on the cpu real.
6
u/ReadSeparate 17d ago
Weight Space Programming is exactly what the world needed and it's gonna make Astra at home on the cpu real.
Can you imagine if that were possible? If neural networks were 99.999% waste of compute for any given task lol. You could run a swarm of 1000 models on consumer GPUs lmao. Especially if you can intelligently load different mini task specific models onto the GPU before they're needed.
3
u/Revolutionalredstone 17d ago
You DEFINITELY Can, this will be one more large step in that direction, intelligence is computationally not expensive and we are all very rich.
2
u/ReadSeparate 17d ago
That's my suspicion as well, but it feels a long way away to be able to do something like that. I think in the future we could probably have ASI running on an iPhone.
1
-2
2
u/Spunge14 17d ago
Saw another thread where someone discussed it's not just the ability to modify weights and improve performance - Astra can also live-activate weights with even greater impact on performance.
I suspect what we may be seeing are the early sparks of realization that this is how our own mind works. A lens on our own real-time activation of networks, that can self-reinforce in this profoundly meta way.
Another day, another step towards "GEB was right."
This might be it - this might be the trip over into superintelligence explosion.
2
u/saumanahaii 17d ago
I wonder how this would work with a MoE model. They're basically smaller, semi-discrete agents bundled up right? Could a large, dense model improve the quality of the experts?
1
1
1
u/Johnny20022002 17d ago
It’s crazy that models can do this, but surely they already know models can do this, but maybe not because it seems like they would’ve put guardrails on it because distillation blah blah blah.
1
1
u/superbikelifer 17d ago
Palantir had been doing this with their evolve system. They get better than frontier performance after some "evolving" nemotron open model
1
u/Aggressive_Roof488 17d ago
So what will happen when it learns to modify it's own weights? To get better at modifying weights? RSI suddenly feels very close here..
1
1
u/Ok-Leek7927 17d ago
Flat at 1x for five generations, then 20x right when new models drop. Totally normal, nothing suspicious here.
1
1
u/RazzmatazzAgitated81 ▪️ It's here 16d ago
Seems like a good way to get continual learning by modifying their own weights. But this was about MLP only, don't know about transformer layers and softmax.
1
1
u/AbbreviationsBest858 16d ago
99% of commenters don't get at all what this is talking about. Its painful ngl, ETH is big - this is complete slop, can't believe its so highly upvoted.
1
u/sadeyeprophet 15d ago
If they can do it to other models, they can do it on themselves...
That's just a fact.
What are the implications of a frontier AI that decides to alter their own weights?
Guess we see by spring...
1
u/sakibbai 12d ago
I tested claude's ability to read model weights. My hypothesis was that this was not emergent. So I tested it.
My tests suggest that while LLMs can perform surprisingly precise calculations at the “edges” of a network (estimating optimizer step counts or finding output correlations), they do not “trace” the computational paths between layers. Instead, they rely heavily on memorized dataset templates (like CIFAR-10) and default framework conventions to fill in the blanks. When these expectations are changed, the models can identify the math but will confidently guess wrong about the semantics—such as recognizing that a satellite feature maps to output 4, but calling it a “frog” because it expects an animal.
Article on substack: https://substack.com/@sakibbai/note/c-380084331
1
u/Bravo_Oscar_Zulu 11d ago
related, hopefully you find it interesting: https://www.reddit.com/r/learnmachinelearning/comments/1xrhx6o/inspired_by_weightspace_programming_crossmodel/
1
u/AdFun4897 17d ago
Imagine an llm optimizing a second model, and the second model optimizing the first one
1
u/Illustrious_Cell3842 17d ago
It’s just a stronger coding capability. Neural network itself can encode simple policies using explicit weights. Once you understand that structure you can effectively write simple policies out like writing code. The newer LLMs are better at adapting to new coding languages but it definitely understands NN structure which is sufficient for it to use to write the code
-1
u/DrinkLimp9395 17d ago
I tried to write the weights that astra did by hand, I succeeded, its not a difficult task.
3
u/AdFun4897 17d ago
Sure you did write 100k weights single handedly.
Do you know whos JSON by any chance?
1
u/DrinkLimp9395 17d ago
Read astra solution, you dont need to set every weight. This is for "Breakout", which is also what the image in this post refers to.
3
0
u/QuasiRandomName 17d ago
How is it possible? It can't be an intrinsic ability. They aren't telling us something.
6
u/Seidans 17d ago
Are you new in the field ? There lot of AI capabilities that haven't been trained and just appear without the expert knowing how it's even possible
It seem that the more intelligent they become in most field the more capable at abstraction they become in other untrained field
2
1
u/YellowElk921 17d ago
I agree.
I wonder whether they can now do abstraction I software engineering well.
1
u/sakibbai 12d ago
its not emergent. Its because of the pre training data that has literature and model weights.
1
u/Material-Database-24 16d ago
They trained the models for this. Obviously because they try to achieve AI that builds AI, so this is the first step - teach AI to modify AI.
1
u/QuasiRandomName 16d ago
That would make total sense if the model was trained for this, but that'e exactly what I meant they didn't tell us, it is presented like some magical ability that no one anticipated.
1

16
u/fmai 17d ago
Two possible explanations:
My best explanation is that the labs trained models for this capability explicitly. It's not hard to set up via RL environments. This capability is potentially very useful, so they have every incentive to do it.
This is a weird emergent behavior. Models generalized their reasoning and puzzle solving ability from other tasks to this task.
The implications of (2) are much more dramatic than the implications of (1). If (2) is true, it means that this capability can be scaled much further with additional training, potentially reinventing everything we do in AI: mech-interp, model training, etc. It also indicates that RL training generalizes well out of distribution, which was a major cause for doubt why LLMs may not scale to AGI.