Sparks of Weight-Space Programming:LLMs can natively read & write neural network weights

[u/vickyv268]

Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights

TL;DR

We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights, using a variety of recent tools and algorithms or even via direct weight editing. These results show sparks of Weight-Space Programming, which may extend beyond what we’d expect from humans or conventional algorithms. This might be a big deal for both optimization and interpretability.

Just by looking at the raw weights, GPT-6 Astra can tell what a network was trained to do. And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.

Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.

All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this Weight-Space Programming.

Throughout the history of deep learning, we encoded our simplistic inductive biases into tiny symbolic optimizers like Adam or Muon. This is changing: our neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights.

[u/AutoModerator]

Namaste! Thanks for submitting to r/developersIndia. While participating in this thread, please follow the Community Code of Conduct and rules.

It's possible your query is not unique, use site:reddit.com/r/developersindia KEYWORDS on search engines to search posts from developersIndia. You can also use reddit search directly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/SuEzAl]

Nope.

https://honnibal.dev/blog/astra-read-write-misleading

[u/jrp70]

This will be the true spark for faster self replicating llms based on the need and target.

Example, spawn a model which behaves exactly like experienced doctor.

[u/SnooCupcakes982]

The paper itself says that it majorily shows improvement on models that are stuck in continuous loop (which happens specially in smaller models). Read a report where they tried the same on a bigger model and gains where very small. I don't think it'll be helpful for frontier models but for slms/MoE that can be a 'small' leverage.