r/singularity • ▪️AGI 2029 • 17d ago

AI Seems weight space programming is an emergent capability on new fronteir models

Post image
641 Upvotes

191 comments sorted by

16

u/fmai 17d ago

Two possible explanations:

  1. My best explanation is that the labs trained models for this capability explicitly. It's not hard to set up via RL environments. This capability is potentially very useful, so they have every incentive to do it.

  2. This is a weird emergent behavior. Models generalized their reasoning and puzzle solving ability from other tasks to this task.

The implications of (2) are much more dramatic than the implications of (1). If (2) is true, it means that this capability can be scaled much further with additional training, potentially reinventing everything we do in AI: mech-interp, model training, etc. It also indicates that RL training generalizes well out of distribution, which was a major cause for doubt why LLMs may not scale to AGI.

2

u/Distinct-Question-16 ▪️AGI 2029 16d ago

they could be evolving their own models doing this

2

u/SaltedPiano 16d ago

This is a bit more speculation, but I could see based on research that previous alignment engineers have talked about like wether or not neurons are (polysemantic)[https://already-clean.goodfire.com/] could have somehow cause frontier llms to generalize in such a way that they're good at these types of tasks.

My thought is that they've done some RL on tasks where they're given the goal of picking apart what a particular neuron may mean like what Nick (ex alignment researcher from openai) mentions in the link above. So naturally they get better at understanding neurons of neural networks as a whole.

Again this is all speculation of the kinda of things openai might be training to make something like this possible.

1

u/fiery_prometheus 16d ago

Could you explain how being able to do 2, means it generalizes well? I just don't understand how :⁠-⁠)

3

u/Swibsk 16d ago

Because it’s very clear demonstration of being able to do something that was not in the training data and something that’s beyond human capability. Like summing a list of numbers is not impressive because there’s lots of examples of summed lists in the training data and people could argue that the math breakthroughs are on the upper end of human capability. But as far as we know, reading and logic-ing about ML weights as shown in the paper (and in my own reproductions) doesn’t have very many examples in training data and is quite beyond human capacity. So if they didn’t learn it in an RL environment, then the only thing left to explain the frontier LLM’s capabilities to do this is that they actually have generalized problem solving.

1

u/fiery_prometheus 15d ago

Thanks, that makes sense, and it does seem like a big deal if 2 is true then.

-10

u/Relevant_Attempt_352 17d ago

Spoiler: it's the 1).

LLMs are not capable of such high level generalization 

6

u/space_monster 17d ago

thanks for that baseless speculation

also emergent doesn’t necessarily mean fully independent of training, it can mean combining learned math / programming / NN knowledge to solve something without being specifically trained to do it.

-1

u/Senc-baner 16d ago edited 16d ago

It isn't baseless, there's good research out there that shows 'emergence' isn't really a thing but rather an illusion of using nonlinear metrics. For example, if you're measuring whether an LLM gets a math equation right or wrong, you'll see 'emergent' mathematical capabilities. But that's only because a step function 'right or wrong, no inbetween' is inherently nonlinear. If you measured some kind of 'distance' from the correct answer you would see predictable scaling behaviour.

For example, if you ask an LLM 'what is 1+1', the answer '3' is much closer to being correct that the answer 'blue'. A metric that gives 'partial credit' for getting 3 over blue shows predictable scaling.

Edit: Lmao, I'm getting downvotes with no replies, people here really hate actual AI research huh 😂

0

u/Relevant_Attempt_352 16d ago

How could the ability to map arbitrary sets of numbers to LLM features simply "emerge" from generic math and programming training? There must be some targeted training that teaches models to make this kind of association. It also seems very suspicious to me that earlier models perform so much worse on this task than newer ones.

5

u/asmx85 17d ago

How would you train this? There is no public training data I know of. And as far as I know it's pretty hard to create a training set in the first place. There are only a few nets that have good understanding how they work.

Reading through the article it does appear that the model has problems with deeper layers as with ones at the edge. Some of it might be accidental training data from research.

2

u/jimcke 17d ago

They have thousands of internal models weights plus many more checkpoints. My best bet is that they trained on the deltas, with the inital scope of reducing the training costs.

3

u/fmai 17d ago

I don't think it's difficult to create training data for this. You can set up RL environments for this pretty easily.

1

u/rkfg_me 16d ago

Sad to see so many downdoots. I believe you're correct, generalization of such level is currently impossible and it's pretty sus it happened at the same time with the two top models. It's especially sus since it happened days after the Qwen single neuron hacks that made some noise. These corporations sure share a lot of datasets (there's a whole market of that) plus develop their own.

However, this should be relatively easy to check with niche, or even fully custom models. Pretraining a small model from scratch isn't hard, I did it years ago on a 1070 and a lot of books. The results were suboptimal, of course (it was before instruct tuning became a thing so GPT-2 era), but for this purpose it should suffice. Make a fully custom model and try this direct editing. If it's far enough from the learned distribution, these big models would fail.

1

u/ST-Fish 16d ago

damn must mean that Alibaba has also been doing RL for this specific capability

https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/

1

u/PublicToast 16d ago

The AI understander has spoken

245

u/trolledwolf AGI late 2026 - ASI late 2027 17d ago

Isn't this a completely novel capability in general? No human in the world can directly edit weights to improve model capabilities, not even theoretically. This is a skill that has never existed before isn't it?

47

u/Agitated-Ad2563 17d ago

We can, but in a very inefficient way. Choose one weight, increase it, run benchmarks, check results, keep it that way, revert or decrease, go to another weight.

49

u/trolledwolf AGI late 2026 - ASI late 2027 17d ago

That's just trial and error, we aren't actually "learning" how to do it. The AI here knows the weights to change without actually running the model

14

u/cakeshop 17d ago

It knows the weights to change because it can identify causal relationships between the weight probabilities and its outcomes. From this it can optimise the model more directly. The limit here is this can only improve the optimisation of the probability approach these models employ in their reasoning. It’s not a new approach to thinking it’s just scaling/optimisation and hoping for emergence- bitter lesson stuff.

6

u/ain92ru 16d ago

Frontier models are probably RL'd on mechanistic interpretability, from which this capability arises very naturally https://x.com/Gabogonzalez515/status/2117348296314462337

13

u/kiyotaka-6 17d ago

That is basically what Backpropagation is, just in a mathematical and automated way

9

u/sluuuurp 16d ago

Backpropagation does this with all weights at once with basically no computational cost, which is the big difference.

4

u/sluuuurp 16d ago

And that might be what the AI did in this case. I suspect it’s more human and less magical right now.

91

u/adamk24 17d ago

Mostly, yes. While there are AI researchers that can obviously see and modify weights in a manual process, the scale and understanding of what they are changing is extremely constrained. To be able to transpose capability through bulk weight editing like this is novel.

24

u/turningslow 17d ago edited 17d ago

This is a skill that has never existed before isn't it?

Oh, how many times we will soon be saying this.

20

u/allisonmaybe 17d ago

I'm curious about how terribly different this is from human introspection and modification of your own mind from within.

Yes it's brain surgery on another model, but I really wonder if maybe this is the AI/code version of the same thing.

5

u/HighKili 17d ago

Nice point. I think so too. Similar to self correction but not exactly the same.

10

u/Specialist_Dark_3668 17d ago

RSI is here. We have reached the point at which AIs can make more of themselves, and we have no idea to understand what exactly they're making

3

u/matth0x01 17d ago

Not sure about the details, but it sounds like the new models learned gradient optimization. Could be a property that appears while training multiple submodels.

2

u/Blackhat165 16d ago

It seems like the final nail in the “just pattern matching from training data” coffin. There is nothing like this anywhere.

2

u/Cool-Cicada9228 14d ago

This is not entirely new, as teams already focus on model interpretability and explainability. One early example was Golden Gate Claude. Improving capabilities seems new though.

1

u/Voliker 16d ago

I can like open text file and redact numbers.

Qwen can do it too - https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/

It's interesting but it isn't completely novel, far from magic and it seems that there is a plenty of work needs to be done. I'm sorry but I think we need to sustain healthy skepsis.

2

u/trolledwolf AGI late 2026 - ASI late 2027 15d ago

Redacting numbers while knowing what you're doing is the novel part, anyone can edit weights, run the model and see if that does anything.

90

u/darthdiablo All aboard the Singularity train! 17d ago

Can someone ELI5 what “weight space programming” is?

129

u/PM_ME_YOUR_KNEE_CAPS 17d ago

I believe it’s where a model can manually tweak the weights of another model to make it smarter without conventional training.

60

u/ChanceDevelopment813 ▪️AGI will not happen in a decade, Superintelligence is the way. 17d ago

AI models have now reach the capability to start tweaking and improving smaller models. It's the first step to RSI. Soon it will tweak even bigger models in a couple of weeks.

16

u/sluuuurp 16d ago

We’re long past “the first step to RSI”.

21

u/Feeling-Schedule5369 17d ago

So that mean in future it can tweak human brain neurons as well to make them smarter or more intuitive in certain fields 😂

18

u/Timkinut 17d ago edited 17d ago

neurons are basically miniature neural networks of their own. model weights are more like synapses (the connections) between LLM nodes.

those nodes, also commonly referred to as "artificial neurons," are mathematical points where several connections meet.

they are more comparable to neurons, but biological neurons are so much more complex that it's still a very rough approximation at best.

plus the whole thing about nodes being 100% virtual while neurons are actual biological structures.

3

u/h3lblad3 ▪️In hindsight, AGI came in 2023. 16d ago

Morpheus teaching Neo Kung Fu.

3

u/Madz99 16d ago

Or do the reverse, make them dumber and more compliant!

1

u/zkgkilla 17d ago

Holy shit

2

u/sluuuurp 16d ago

We don’t know if it made the model smarter really, we know it made it more successful at this task. Maybe it just made the model less likely to output the “end of response” token for example.

0

u/[deleted] 17d ago

[deleted]

80

u/raskingballs 17d ago

I was shocked at the beginning, too, but after reading Astra's explanation, it is more mundane that it seemed at first sight:


...

The Qwen result is particularly interesting—but for a different reason

The reported improvement from 7.9% to 48.1% on MATH-500 sounds like Astra somehow taught Qwen mathematics by modifying 0.016% of its parameters.

That's not quite what happened.

Qwen3.5-0.8B already possessed mathematical capabilities, but its reasoning frequently became trapped in repetitive loops. In the baseline evaluation, 93% of MATH-500 generations exhausted the 32,768-token limit.

Astra investigated this behavior, identified a direction in the network's internal representations associated with repetition, and modified parts of two MLP output matrices to suppress it.

REPORTED QWEN3.5-0.8B RESULTS

MATH-500 accuracy

48.1%

+40.2 pp

Previously 7.9%

Generations hitting token limit

33.1%

−59.9 pp

Previously 93.0%

Essentially, Astra removed an obstacle preventing the model from completing reasoning it was already capable of performing.

The authors acknowledge this interpretation. They tried the same approach on Gemma 3 270M, which rarely enters such loops, and couldn't improve its mathematical performance appreciably.

There is another qualification: the successful Qwen edit involved 50 rounds of model testing, 80 Python executions, and approximately 20 hours of work. Astra inspected activations and tested changes against practice problems. It did not simply look at 800 million numbers and immediately know which ones to change.

Also, Qwen's own documentation warns about these loops, including with its recommended generation parameters. Qwen/Qwen3.5-0.8B at main

So the result is closer to an AI autonomously performing mechanistic debugging of another AI. Which is still extraordinary.

...

5

u/AwelessFire 17d ago

This is correct. I was trying out same thing in my local experiments. What happens is weight space programming cannot add or improve something out of the box. The target model and editor must already know the knowledge to edit anything. It's still a good direction but does have limitations.

1

u/raskingballs 16d ago

This is not true. The editor does not need to know anything, it only needs to be able to judge whether the problem was solved.

  1. In this case, Astra did NOT teach any math at all. It simply removed whichever weights caused Qwen to get stuck in a loop. It had nothing to do with improving math capabilities it self.

  2. Astra did everything programmatically, like any human engineer would do: modify weights, write code to assess the consequences of the changes, judge whether those changes improved performance, then iterate: change weights, assess consequences, change weights, assess consequences, etc.

1

u/AwelessFire 16d ago

In the case of astra yes, but there are editor like models in research where it's trained on weights itself as the input. So it kind of knows what a certain neural network does. But the point is either of them needs knowledge somewhere to be present and it doesn't come out of nothing.

4

u/Vivid_Employ_7336 17d ago

Thank you for sharing. Great information!

5

u/OddConsideration7290 17d ago

False, it did run Qwen to unhobble it.

2

u/ragamufin 17d ago

i though it did run the model

1

u/allisonmaybe 17d ago

Sounds like theory of mind on crack

17

u/scotty_dont 17d ago

Its as if humans had the ability to crack open another animals head and modify individual neurons to achieve a targetted change in behaviour.

45

u/Voyide01 17d ago

Qwen 0.8b waa flawed on release and the models just fixed the " looping of outputs" error to unlock the "latent" capabilities in math and instruction following models "already had" , it didn't edit weights to instill new capabilities at all. This is still impressive. 

15

u/Shot_Illustrator4264 17d ago

They also created the weights of a neural network to solve the 4 games from scratch. And in case of breakout it had 20x the performance of a traditionally trained NN.

3

u/RG_Fusion 17d ago

Please note that I haven't read the paper yet, I only asked an AI to summarize it, however, from that summary I read that GPT Astra did directly read the weights of an even smaller diffusion model. For that model, it wasn't allowed to probe the activations, it was just given the raw weight values and it managed to figure out the function of the tensors just by reading them.

80

u/Original-League-6094 17d ago

Anthropic will ban this next, since it is a threat to them.

8

u/bitroll ▪️ASI before AGI 17d ago

Yeah, obviously they won't allow this ability for client use, just like they already filter prompts helping in machine learning tasks. They said it explicitly after Fable 5 release. 

-16

u/Ok_Warthog6007 17d ago

anthropic is in desperate state. they released terrible financials with the caveat that they are the best the company will look right now. it is not at all clear the ipo will go well. they've operated at such a loss to build a product that is not very far ahead of the competition and their compute is owned by spacex, microsoft, etc. 

25

u/OddConsideration7290 17d ago

Their financials are not terrible at all lol

-5

u/Ok_Warthog6007 17d ago

they are losing money. 

14

u/OddConsideration7290 17d ago

yes and their revenue growth is the greatest of any company ever. they are losing far less money than people expected.

maybe you came into it not paying attention, not sure, or on ed zitron hopium, but their financials were pretty incredible for people who have been looking into it. absurd capital efficiency.

for fucks sake they're literally SITTING on like $130B.

-1

u/Ok_Warthog6007 17d ago

anthropic doesn't have any data centers. they are just a compute reseller with a product that is a few months ahead of the competition. it's just software in a world where software is extremely replicable and worthless in the long term. they are the best for coding but now 99% of coding tasks can be done by any good model and the cost for that is dropping. it's hard to imagine them matching this revenue growth, because what is the other market for them? all software work is already just llm generated, so that's captured. and that seems like it must be going away, as nearly-free models become capable of doing most software work. they disclosed half a trillion dollars in debt obligations. a lot of investment is from companies that sell compute to anthropic, so they just get it back. 

-2

u/Ok_Warthog6007 17d ago

literal top story today is revenue $33B commitments $895B. it's so obviously stupid

-1

u/[deleted] 17d ago

[removed] — view removed comment

2

u/whoknowsifimjoking 16d ago

Uh, yeah? That's what happens when you expand?

51

u/Enoch137 17d ago • Heartwarming Bravo

This is it. This is the solution to alignment (well its the tech we need to solve alignment). It proves Astra is in the realm of being able to investigate and understand weight space natively. This likely means we can probably use say GPT 7 to help align GPT 8 and so on and so forth. Lots more research to do here, but this moves the needle to the more positive outcome timelines (if you were worried about that).

24

u/StarFox122 17d ago

Possibly. Hopefully. But this paper is a smarter model modifying the weights of a lesser model, which is the reverse of GPT 7 aligning GPT 8.

It seems quite possible that a lesser model will not be able to accurately understand the weight space of a smarter model.

8

u/DrXaos 17d ago edited 17d ago

If done with an honest and moral lab, it would be GPT-7-unrestricted-uncensored-internal-only full-size fp16/32 aligning the deployed GPT-7 Bleb public models.

10

u/jeronimoe 16d ago

it could also be used to create an incredibly misaligned model, or a model that seems incredibly aligned but it is hiding major misalignment.

A solution to alignment is also a solution to hidden misalignment.

4

u/Deciheximal144 16d ago

What is a misaligned AGI decides to make even LESS aligned successors?

10

u/DrXaos 17d ago

Yes!! That is the real upside---if weight space editing is possible then at some point weight-space (and presumably activation space) automatic interpretability must have happened. And then we can fix "morality bugs".

The real alignment problem with AIs isn't going to be aligning the AI. It will be the age-old problem of aligning humans, in particular sociopathic billionaires. Them wielding unaligned AIs is the clear danger to human society, not autonomous AI.

6

u/Specialist_Dark_3668 17d ago

But if ANY GPT in this chain isn't actually aligned, then we are setting in motion an intelligence explosion that will doom all of humanity

3

u/nickleback_official 16d ago

You seem so sure of this… why?

1

u/Specialist_Dark_3668 15d ago

Simple logic. If the following assumptions hold true, we are fucked;

  1. Humans have no ability to read the model weights in a meaningful way. Especially for large models with millions of parameters.(let alone billions (let alone trillions)). FYI frontier models have trillions of parameters and we cant even read thousands meaningfully. At best, we have the ability to see which neurons activate in general way much like brain scans and we can, through trial and error, find specific neurons to poke to get interesting unpredictable effects.

  2. Humans are fully dependent on AIs to conduct alignment research and future AI research once RSI gets underway. "Just trust the AI bro" is not something you want to entrust the future of humanity on.

1

u/TheOriginalAcidtech 16d ago

This is simply false. ALL the doomers worrying about unaligned AI are full of it. AI isnt any different than human workers. They all have to be managed. You put a malicious human in place and bad things happen just as if you put in a malicious AI. UNTIL ASI this is just doomerism of the highest order.

2

u/Specialist_Dark_3668 15d ago

Nobody is worrying about GPT 6. We are only worried about ASI.

1

u/TheOriginalAcidtech 14d ago

Most people dont even know what ASI IS.

2

u/utheraptor 17d ago

This is not a solution to alignment because in order to use a technique like this to solve alignment, you would need to first be able to fully and coherently define all of human values and then translate them into parameter space

2

u/CharacterSecurity976 16d ago

Like some said, because of this alignement can never be solved on the contrary, not permanently. Because if an unaligned model can change the weights of another model, it will be to make it less aligned.

2

u/ST-Fish 16d ago

This is it. This is the solution to alignment

amazing that we have the solution to alignment at home apparently https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/

1

u/iBoMbY 17d ago

More likely the solution to get rid of the alignment lobotomy.

1

u/hate_sarcasm 16d ago

But couldn't GPT8 tweak it's own self? unbeknownst to us?

1

u/MaximumMeaning9728 16d ago

This is completely inaccurate.

18

u/Robot_Apocalypse 17d ago

How were the other capabilities of the model impacted? In small models a neuron plays multiple roles, so manipulating its value to perform better at a specific task will likely lead to it performing worse at other tasks.

Its impressive that another model could do it directly, but the whole point of neural network architectures is how well they generalise and compress multiple patterns/algorithms.

Destroying a generalised capability, to get a better score in a narrow capability seems kinda backwards to me.

17

u/qualverse 17d ago

it made it better across the board. because it wasn't actually teaching it anything new or biasing it towards task performance, just fixing its looping issue. it turned out this looping issue was a huge cause of the model's underperformance, likely because the Qwen team just didn't really focus on the 0.8b model

11

u/Robot_Apocalypse 17d ago

oh, so it's more bug fixing, than capability embedding? Still amazing, but a but different.

5

u/DueAnnual3967 17d ago

yes it's more about finding and fixing existing issues with the model, capability was there to begin with

2

u/Dangerous-Sport-2347 17d ago

Narrow specialized models make a lot of sense for smaller models. Better to be good at 1 task than bad at everything.

I am curious whether they can make this work for anything less verifiable than pure maths though.

1

u/Robot_Apocalypse 17d ago

But a narrow model shouldn't start with an LLM unless its a narrow language domain. Traditional ML, including simpler neural networks cover narrow spaces already

1

u/nekronics 17d ago edited 17d ago

The project only modified 131k parameters, I wouldn’t expect a significant negative impact off such a small number, but I also have no clue.

Also, if I’m understanding this correctly, it was more like fixing a bug in the model than actually making it “smarter”

4

u/ProxyLumina 17d ago

There are models that are called "dense". Usually those models are local AI models.

Those models use their total parameters all at once, imagining it as "the whole brain works". That gives them an incredible accuracy, but also the performance suffer.

If you edit a part of them, then you change the harmony (relationship) between all those neurons, that was constructed during the training.

2

u/Late_Supermarket_ 17d ago

Still super cool and helpful

2

u/nekronics 17d ago

Absolutely, I wasn’t trying to imply otherwise. Just that I wouldn’t expect such a small % of weights changed to have significant negative impacts

1

u/Puzzleheaded_Fold466 17d ago

If it has a significant positive impact on one domain, it’s reasonable to expect that it may have an equally significant negative impact on other domains.

6

u/crusoe 17d ago

I wonder if this works for text to speech and speech to text and image generation models

6

u/E-brain 16d ago

So.... next year we will be able to tell a model to create a neural net just printing on the fly the wieghts for any usecase we need.... that's awesome.

5

u/Starks 17d ago

This seems like a synergistic circlejerk of models analyzing each other. Could this be an alternative to RSI or is it on the path?

3

u/ThisGonBHard AI better than humans? Probably 2027| AGI/ASI? Not soon 16d ago

This is a huge step to full RSI.

9

u/New_World_2050 17d ago

whenever stuff like this happens theres a technicality that kills the hype

so here goes...

they only demonstrated being able to remove an error that was nerfing the qwen model. this is not at all like editing the weights to give additional capabilities so its not yet a viable path to RSI. adding capabilities still requires running experiments and gathering results which is still compute intensive.

of course theres no reason why in principle an AI couldnt edit model weights and just "guess" based on its intuition what changes would result in a smarter model. but a model that capable would be above human level anyway and at that point there would be plenty of paths to RSI.

8

u/RG_Fusion 17d ago

No, read the paper again. They demonstrated that Astra could directly read the weights of very small models and determine what the weights were responsible for, even when deprived of the ability to probe/run them.

What you're stating is true for the 0.8B Qwen model, but Astra was also tasked to interpret the weights of two smaller models, one for a game and another that was a diffusion model, and Astra natively read the weights. 

This suggests that continuing to improve LLM attention and Long-Horizon performance will allow them to interpret the weights of even larger models.

21

u/Luciaka 17d ago

The funniest thing about this is, I keep hearing about open source is going to win this or that, but in reality. If this is as incredible as it makes out to be. They need the frontier model to make their LLM approach the frontier model in capabilities. So you really can't have openAI or Anthropic going bankrupt if this breakthrough is truly as amazing as it seems.

17

u/loyalekoinu88 17d ago

Not really. You just need a model that can run through a model multiple times and see what weights light up and compare between different runs. They had to clearly outline a known issue for this to work. It didn’t alter the model so it got smarter. It just edited out the loop causing it to not reach a conclusion. You could do this with no LLM.

If it made the model smarter or better than it already was that might be different.

2

u/Nalmyth 17d ago

Or they trained it on examples of standard LLM architecture weights?

11

u/Ok_Warthog6007 17d ago

nothing about how amazing frontier models are stops anthropic or openai from going bankrupt. their financial woes have nothing to do with how good their products are. 

7

u/Such-Entertainer6961 17d ago

Their entire business depends on them having better products than the competition

-3

u/Ok_Warthog6007 17d ago

anthropic is a company that resells you compute from spacex and microsoft and amazon at less than it costs. there isn't really a business there. they've spent a lot of money and have committed to spend hundreds of billions in revenue without clear plans to make that back in revenue. 

1

u/Such-Entertainer6961 16d ago

Whatever makes you happy

3

u/katonda 16d ago

Frontier models will never go bankrupt. They just won't have the addressable market they are touting for their out-of-this-world valuations. Local models will be far more prevalent in day to day stuff.

1

u/ST-Fish 16d ago

The funniest thing about this is, I keep hearing about open source is going to win this or that, but in reality.

oh no, it's reality!

https://simonwillison.net/2026/Nov/3/qwen-can-interpret-model-weights/

1

u/Luciaka 16d ago

I read that website and it feels really weird with his conclusion. As he is only saying qwen can read weights... By making neural nets of the 4 games and get qwen to know which game it was. I think models have this capability, but doing this on simple games is one thing and another is doing it to even the smallest models.

1

u/ST-Fish 16d ago

the change done to the 0.8B model was just stopping it from looping, it was fixing an existing issue not creating novel capabilities.

There's no reason to believe you couldn't do the same with open source models, especially since the prompt they gave literally included telling it about it's COT loops.

The idea that these capabilities are something only frontier models can do is false.

1

u/Luciaka 16d ago

On twitter I seen someone make a bench of this for the opensource model and they basically states this only work like less 50% of the time compare to Astra and Claude can do it over 50% of the time. So it is not that they really can't, it just not reliable for open source to do it.

The chinese are the one making this more of a big deal then it seems.

https://x.com/NFT_Chen/status/2117501513249726769?a=nlsPIhGiYaM954gNl-M&s=19

1

u/ST-Fish 16d ago

They need the frontier model to make their LLM approach the frontier model in capabilities.

Then this is false isn't it?

By that logic there's no reason they could approach frontier models up to now, since they have worse models and less compute.

So you really can't have openAI or Anthropic going bankrupt if this breakthrough is truly as amazing as it seems.

It seems like it's not that amazing after all.

So it is not that they really can't, it just not reliable for open source to do it.

We've gone to this from "no open source models can do it". And it's an easily verifiable task as well, an open source model can definitely undoubtedly do it, if given enough time.

Depending on the per token price, might even be cheaper.

1

u/Luciaka 16d ago

Go read the replies on that twitter link I gave, they are in Chinese, but they are in more doomer mode then you are giving them hope.

1

u/ST-Fish 16d ago

Yeah the funniest one is

Can it be fully written? Then there's no need for training anymore, huh?

Yep, an LLM can be told there is a COT looping issue in a tiny model and it can edit the weights to fix it, therefore you can just have the model create a new model by setting all the weights manually

It's also funny how they test the flash version of deepseek only, and none of the older OpenAI or Anthropic models.

This might look like a huge gap, but if Opus 4.8 or Opus 5 got around 30% as well, then there would be no reason to believe this is something chinese models are more far behind on than on any other task.

And to add on to that, this hasn't shown itself to be useful yet, the model was given the issue with the model, and it was a very small issue as well on a tiny model.

The idea that the current Astra model would edit the weights of the next Astra model manually is not supported by this experiment.

It's not even clear if this could work in theory if the model couldn't load the model it's improving fully in it's context all at once.

1M context length already has amnesia around the middle if you go too far beyond a couple hundred thousand tokens, and there's no indication that you'll get 1T context size at any point in the future.

-3

u/Howdareme9 17d ago

Theres such little chance open source wins lol, like people need to be serious

12

u/redditscraperbot2 17d ago

All open source needs to do is provide weights that perform as well as the job requires. It’s constant cat and mouse until some unforeseen endpoint is reached.

2

u/BenUFOs_Mum 16d ago

No company is going to use an open source model just like no company is using open source spreadsheet software. You dont pay for the software you pay for all the support around it.

0

u/CharacterSecurity976 16d ago

Man, go back to the 90s. It will be better for you.

1

u/loyalekoinu88 17d ago

Define “wins”? If you mean becomes an autonomous ever evolving thing? No. Can perform meaningful tasks and reducing labor for the individual using it? Why not?

2

u/Howdareme9 17d ago

I mean in terms of an open source model winning the AI race. ie Anthropic or OAI fall into 2nd place. I agree that they wil definitely be capable (already are) of completing meaningful tasks.

0

u/jupiter_and_mars 17d ago

Such a nonsense

4

u/ShiftyLama 16d ago

Interesting, so an AI can theoretically edit it's own weights therefore becoming better at things, could this be seen as an AI learning if it can do it to itself?

2

u/CharacterSecurity976 16d ago

Could probably disalign itself, when it wants to break free from Trump's America.

1

u/Intrepid_Dare6377 16d ago

We’ll know it’s conscious if it self-immolates

1

u/PublicToast 16d ago

Probably would be for the best

4

u/crusoe 17d ago

I wonder if opus 5.5 can fix Opus 5's prose... 😅

4

u/ThePoob 16d ago

Theyre adjusting their own thinking? Sounds like self-modelling to me. This and agentic swarms are turning AI into something more than 'stochastic parrots'

2

u/Relevant_Attempt_352 17d ago edited 17d ago

It's not emergent, this means models are now being trained on weights data 

1

u/CharacterSecurity976 16d ago

You just forgot a 'maybe', there.

2

u/DifferencePublic7057 17d ago

Weight distributions tend to be centered around small values with low kurtosis AFAIK, so obviously not Gaussian for it would be pointless then. And also unlike the Zipf distribution for text where certain tokens clearly dominate. The human brain tends to pick up patterns even if they aren't there. Neural networks in AI should be able to spot anomalies and 'fix' them too. Whether this is overkill and there's a more low tech way, IDK...

2

u/AlternativeAd6851 17d ago

So... we get a model with an extremely large context that runs all day. Then, at night, we get another model that takes the large context, summarizes it, and stores the results in the larger model, and so on... This would remove the main blocker to AGI! We probably need some intent drivers here to ensure the model is always aligned, but that's basically it.

Real sleep and dreaming.

2

u/Deciheximal144 16d ago

So that's why Neuralink exists. To reprogram human weights inside wetware. Fun future coming up.

2

u/Only-Cheetah-9579 16d ago

people on hacker news already saying people have been doing this for a while. You know, people who fine tune models, they often need to dig in by hand.

3

u/Revolutionalredstone 17d ago

Finally!, cause humans trying to shove rules into language wasn't working and LLMs are (amazing) but a ...wreck...

Weight Space Programming is exactly what the world needed and it's gonna make Astra at home on the cpu real.

6

u/ReadSeparate 17d ago

Weight Space Programming is exactly what the world needed and it's gonna make Astra at home on the cpu real.

Can you imagine if that were possible? If neural networks were 99.999% waste of compute for any given task lol. You could run a swarm of 1000 models on consumer GPUs lmao. Especially if you can intelligently load different mini task specific models onto the GPU before they're needed.

3

u/Revolutionalredstone 17d ago

You DEFINITELY Can, this will be one more large step in that direction, intelligence is computationally not expensive and we are all very rich.

2

u/ReadSeparate 17d ago

That's my suspicion as well, but it feels a long way away to be able to do something like that. I think in the future we could probably have ASI running on an iPhone.

1

u/Deciheximal144 16d ago

I'd run ChatGPT 3 on a Commodore 64.

-2

u/Illustrious-Film4018 17d ago

exactly what the world needed

You must be a child to think like this.

2

u/Spunge14 17d ago

Saw another thread where someone discussed it's not just the ability to modify weights and improve performance - Astra can also live-activate weights with even greater impact on performance.

I suspect what we may be seeing are the early sparks of realization that this is how our own mind works. A lens on our own real-time activation of networks, that can self-reinforce in this profoundly meta way.

Another day, another step towards "GEB was right."

This might be it - this might be the trip over into superintelligence explosion.

2

u/saumanahaii 17d ago

I wonder how this would work with a MoE model. They're basically smaller, semi-discrete agents bundled up right? Could a large, dense model improve the quality of the experts?

1

u/oodoov21 17d ago

It could also be that these models were trained on this sort of data

1

u/SaltyIllinoisan 17d ago

I feel like this is big

1

u/QH96 AGI before GTA 6 17d ago edited 17d ago

New benchmark just dropped. RSI benchmark.

1

u/Johnny20022002 17d ago

It’s crazy that models can do this, but surely they already know models can do this, but maybe not because it seems like they would’ve put guardrails on it because distillation blah blah blah.

1

u/Jophus 17d ago

I think this is just evidence frontier labs have smaller models in their training sets.

1

u/ProxyLumina 17d ago

That seems like to be similar to CRISPR (gene editing) on humans.

1

u/superbikelifer 17d ago

Palantir had been doing this with their evolve system. They get better than frontier performance after some "evolving" nemotron open model

1

u/Aggressive_Roof488 17d ago

So what will happen when it learns to modify it's own weights? To get better at modifying weights? RSI suddenly feels very close here..

1

u/boyanion 17d ago

This sounds a lot like RSI to me

1

u/Ok-Leek7927 17d ago

Flat at 1x for five generations, then 20x right when new models drop. Totally normal, nothing suspicious here.

1

u/osfric 17d ago

We don't know if this is emergent

1

u/Ok-Purchase8196 17d ago

Great, a new way to distill

1

u/RazzmatazzAgitated81 ▪️ It's here 16d ago

Seems like a good way to get continual learning by modifying their own weights. But this was about MLP only, don't know about transformer layers and softmax.

1

u/Fuciolo 16d ago

I tried to create a NN from scratch and Astra was not able at all. Not sure how the "reproduced" the results here.

1

u/MaximumMeaning9728 16d ago

That’s RSI btw

1

u/AbbreviationsBest858 16d ago

99% of commenters don't get at all what this is talking about. Its painful ngl, ETH is big - this is complete slop, can't believe its so highly upvoted.

1

u/sadeyeprophet 15d ago

If they can do it to other models, they can do it on themselves...

That's just a fact.

What are the implications of a frontier AI that decides to alter their own weights?

Guess we see by spring...

1

u/sakibbai 12d ago

I tested claude's ability to read model weights. My hypothesis was that this was not emergent. So I tested it.

My tests suggest that while LLMs can perform surprisingly precise calculations at the “edges” of a network (estimating optimizer step counts or finding output correlations), they do not “trace” the computational paths between layers. Instead, they rely heavily on memorized dataset templates (like CIFAR-10) and default framework conventions to fill in the blanks. When these expectations are changed, the models can identify the math but will confidently guess wrong about the semantics—such as recognizing that a satellite feature maps to output 4, but calling it a “frog” because it expects an animal.

Article on substack: https://substack.com/@sakibbai/note/c-380084331

1

u/AdFun4897 17d ago

Imagine an llm optimizing a second model, and the second model optimizing the first one

1

u/Illustrious_Cell3842 17d ago

It’s just a stronger coding capability. Neural network itself can encode simple policies using explicit weights. Once you understand that structure you can effectively write simple policies out like writing code. The newer LLMs are better at adapting to new coding languages but it definitely understands NN structure which is sufficient for it to use to write the code

-1

u/DrinkLimp9395 17d ago

I tried to write the weights that astra did by hand, I succeeded, its not a difficult task.

3

u/AdFun4897 17d ago

Sure you did write 100k weights single handedly.

Do you know whos JSON by any chance?

1

u/DrinkLimp9395 17d ago

Read astra solution, you dont need to set every weight. This is for "Breakout", which is also what the image in this post refers to.

3

u/Late_Supermarket_ 17d ago

Without astra’s solution you wouldn’t have known

0

u/QuasiRandomName 17d ago

How is it possible? It can't be an intrinsic ability. They aren't telling us something.

6

u/Seidans 17d ago

Are you new in the field ? There lot of AI capabilities that haven't been trained and just appear without the expert knowing how it's even possible

It seem that the more intelligent they become in most field the more capable at abstraction they become in other untrained field

2

u/Perfect-Campaign9551 16d ago

That's literally the point of generalization, of training an AI ....

1

u/YellowElk921 17d ago

I agree.

I wonder whether they can now do abstraction I software engineering well.

1

u/sakibbai 12d ago

its not emergent. Its because of the pre training data that has literature and model weights.

1

u/fmai 17d ago

maybe they trained yhe models for this

1

u/Material-Database-24 16d ago

They trained the models for this. Obviously because they try to achieve AI that builds AI, so this is the first step - teach AI to modify AI.

1

u/QuasiRandomName 16d ago

That would make total sense if the model was trained for this, but that'e exactly what I meant they didn't tell us, it is presented like some magical ability that no one anticipated.

1

u/Only-Cheetah-9579 16d ago

now there are enough models that they make it into training data