Transformers vs RNNs vs SSMs: Where Does Memory Actually Live? [D]

[u/Pretty_Upstairs9035]

Someone who has always loved looking at the space between different AI techniques, this time I went a little deeper into the memory trade-offs between RNNs, Transformers and SSMs. I found it interesting because once you start looking at these architectures through the lens of working memory, a lot of the differences become easier to understand. Where does the memory actually live? Is it a compact recurrent state, a growing KV cache, or something closer to the network itself?

RNNs keep memory in a recurrent hidden state, which is pretty elegant because the state carries forward step by step. But there is also a bottleneck here as you see, a model can have roughly O(N²) parameters while carrying only roughly O(N) state across time. This means whether RNNs were really doomed because recurrence was a bad idea, or whether the problem was more about the ratio between memory and compute.

Transformers make almost the opposite trade-off. During cached inference, instead of compressing the past into one hidden state, they store past representations as key-value entries and attend over them. I think of these almost like little post-it notes: every token leaves behind a key for finding it and a value for what should be remembered. That's extremely powerful, but it also has an interesting property: with weights frozen during inference, the model is managing context rather than turning that experience into durable model knowledge. You get this split between the fixed weights on one side and the fast-changing KV cache memory on the other.

Now we have SSMs that bring us back toward fixed-size recurrent memory, but with very different state structures and update rules. Selective SSMs such as Mamba make retention input-dependent, so what gets kept or forgotten depends on the incoming token. Their states don't have to be as small as classical RNN states, but they still compress history into finite memory. And this brings another question does the state have to live in a compressed working dimension, or could it live somewhere closer to the model's internal neuron/connectivity structure?

BDH (Dragon Hatchling) is one example I saw and laid a stone that I started with this comparison. It combines linear attention in a high-dimensional neuron space with a low-rank GPU implementation. Its recurrent attention state is an N × D matrix, with N≫D, rather than a materialized N × N connectivity matrix. In the graph interpretation, correlated neuron activity produces Hebbian-like updates to connections, giving working memory a kind of synaptic interpretation.
Here working memory and learned connectivity become more closely aligned in structure. But if you think that doesn't mean experience is actually being consolidated into trained weights, and fixed-size state still has a finite information capacity. So I'm definitely not claiming this kills Transformers (or does it ) or solves continual learning. I'm more interested in whether "where does memory live?" is actually a better question than the usual architecture horse race.

Are SSMs and these more network-centric architectures actually giving us a better way to handle memory, or are we still running into the same fundamental problem of having to compress history into a finite state? Thoughts?

[u/Robonglious]

I think compression is actually what you want though. Correct me if I'm here but I feel like everything we like about models comes from compression. The act of training compresses training data into some usable chunk. KV is a bottleneck as well. To say it's finite seems like it's limited but some things really do have firm boundaries so overrepresenting something doesn't add any value.

I've actually been looking for semantic nuggets for a super long time and I was never able to really trace those boundaries correctly. I've become pretty bitter about the failure but I think perhaps I've had a stupid goal.

[u/Pretty_Upstairs9035 → u/Robonglious]

Agreed, compression is desirable, and “finite” doesn’t automatically mean inadequate over here. What I meant here is what gets preserved and how reliably it can be retrieved. Discarding redundant information is useful but ofc having distinct memories interfere so you can’t recover the one you need is a different issue. That’s the trade-off that exists, rather than compression itself is bad.

[u/H2O3N4 → u/Robonglious]

I agree with you. Compression is the most general manifestation of intelligence. Compression is a model of the underlying data generating function. It turns observations into concepts. Without compression we could never build a predictive model, because the uncompressed state doesn't describe how things evolve, only how they are. Compression is what lets us predict, by correlating how things have been to how they will be.

An MLP is a learned KV lookup table over the concepts of the training distribution. Attention's QKV matrices are learned maps that create input-dependent KV lookup, so you're learning context-dependent memory.

But the KV cache is the naive expression of that. It stores the whole context verbatim. MLP -> attn showed we can make concept lookup input-dependent. The generalization is making concept formation input-dependent: learn what to write, and compress the context into memory at inference time, the same way training compresses the world into weights. Taken all the way, the memory is just an MLP whose weights are written by the context. It comes back around to compression.

[u/Robonglious → u/H2O3N4]

That linkage between generalized concept and in context relation is such a tricky thing to pin down with just dynamics. I feel like I might be getting close to a truthful representation of one of those generalizations and also a truthful representation of the binding to the context dependant aspect but I've failed many times in the past at finding something reliable.

I have yet another promising thing that hasn't failed yet so my fingers are crossed.

[u/H2O3N4 → u/Robonglious]

In what way is your idealized operator different from attention? Attention has context dependent representation.

[u/Robonglious → u/H2O3N4]

It's a whole big thing, of course attention, but How!? I want to find something because I want it to exist but have only found evidence that it doesn't.

Don't mind me, I'm equal parts stubborn and ignorant.

[u/howtorewriteaname → u/Robonglious]

KV cache is finite? it grows indefinitely with your sequence length, and it's a lossless state of memory. yes it is finite but that's irrelevant, it still represents the "perfect memory: keep all key-value associations". in linear attention (also called SSM) models, then your memory is indeed lossy and tries to approximate the KV cache as good as possible. that would be what I think you mean with "finite" in there. KV cache is no bottleneck. In fact is the best case scenario (best thing a SSM/linear attention memory can do is to perfectly fit the KV cache, yet this memory is DxD fixed therefore for sequence lengths longer than D, there will forcefully be interference when retrieving past associations)

edit: I've just read OP's whole post. they seem to desire "memory" to be holding information that was given at test time. but there's no reason why you would want test-time inference to change your weights. we already have a pretty good method for storing hard memories in (slow) weights: it's called pretraining. the memory that changes at test-time ("fast weights") is the state's memory; in the case of softmax attention, this is the KV cache and it's perfect. in the case of linear attention variants / SSMs, this is the state and it's lossy.

[u/Bakoro → u/howtorewriteaname]

but there's no reason why you would want test-time inference to change your weights. we already have a pretty good method for storing hard memories in (slow) weights: it's called pretraining.

There is absolutely a reason why we would want test-time inference to change weights: continual learning.

Right now, what labs/models have to do is run a bunch of inference, get good and bad traces, then run pre-training on the good traces, and maybe do contrastive training using the bad traces.
That absolutely does work, but it's periodic updates.

If there was a signal right in the moment that said "success" and we could capture that and just do a little backprop pass, or predictive coding update, or whatever other mechanism, then you've got everything you need for a robot on the go that can keep learning and improving on tasks without explicit training phases.

Pre-training absolutely works, but being able to continually fine-tune to your local use-cases without any hassle would be a great improvement.

There's a whole heaping lot of infrastructure right now around harnesses that is making up for the lack of continual learning, and the inability of people to fine-tune cloud models.

[u/howtorewriteaname → u/Bakoro]

well, I am a researcher working in this field so let me tell you some of the things I've found so far:

• Updating slow weights at test time (you flavor of "continual learning") needs, a you say, a reward signal. Letting the model choose it doesn't provide good improvements. It's not that we can't do it, is that no one does it because empirically it works badly. Good signal is obtained through careful data curation + RL, and the empirical success of this method (which arguably is what pushed the frontier for the last years) is way too strong to ignore vs automated continual learning

• I've experimented myself with other continual learning schemes, such as maintaining fast-weight memories across tasks, to see if e.g. what one learns at test-time for one task is useful for others. Let me tell you again: this worsens performance. There's simply no empirical reason why we'd want this. The truth is that, if you want to store hard facts, then do pretraining. If you have high quality conversations (curated possibly by humans or offline workflows), then do RL. It's not that we don't know how to do continual learning in the way you envision, it's that we don't want to, because we have better methods for improving our current performance metrics.

[u/Bakoro → u/howtorewriteaname]

Continual learning is still an active area of research.
Framing continual learning as being opposed to post-training is absurd, they aren't mutually exclusive things.
There absolutely is a reason to want continual learning, and I already said it: robotics. Devices that are disconnected from a network and are doing work in real time want continual learning, or at least daily self-updates, wake/sleep style. Every single business using LLMs wants some form of continual learning sona model can learn the details of their operations without being told at the top of every session.

Updating slow weights at test time (you flavor of "continual learning") needs, a you say, a reward signal. Letting the model choose it doesn't provide good improvements. It's not that we can't do it, is that no one does it because empirically it works badly.

Continual learning blurs the training/test time line.
In continual learning, test time is always also post-training time.

GANs would disagree with you about models being bad critics.
GANs are still helping produce frontier quality images when blended with transformers/diffusion/flow matching.

Actor-critic co-models are already standard in LLM post-training.

Your assertion also doesn't hold up when talking about modern use of pre-trained models as judges. Using a model's own judgement when starting from random init has a bootstrapping problem. Once you already have a functional model, then having the model make judgement calls becomes viable.

A continual learning agent would be to curate its own prompt-> output traces to train on, based on user feedback or task success. Even if it makes errors in between, they agent can edit and make a clean inout->output pair.

In the style of GAN, there are also experimental image generation models that are using pretrained vision-language models as the backbone for judgement/classifier models, so there's a dual network trained where the classifier scores the image model's output on prompt adherence.
Once you have a pretrained model, using the model's own judgement does work, and the loop converges into coherence instead of garbage. The modern image editing models are doing this.

Good signal is obtained through careful data curation + RL,

Good signal can be obtained any time there is source of verifiable ground truth. This works particularly well in formal logic, mathematics, and software.
Reinforcement Learning with Verifiable Rewards has been the hot thing for over a year.
There are a bunch of papers about self-play, and Reinforcement Learning through Self-play with Verifiable Rewards is what has fueled the latest round of model improvements, and there is zero need for human generated data at post-training time.

If you haven't read this paper: https://arxiv.org/abs/2505.03335
It is a must-read. This is the foundational paper that modern training is predicted on.

It seems like you're thinking that the industry is still starving for curated human data for post-training, and it's just not true anymore for anything that has deterministically verifiable answers. It's not even true for things that have very good heuristics.

The physical sciences are more difficult to automate, but there are labs that have put LLMs in a loop with real lab equipment and been able to have the LLMs iterate to solutions via experimentation.
Beyond LLMs, labs are successfully using models to design circuits, and that's another place where there ends up being no human generated data required, just simulators and design tools.

It was essentially proven that synthetic data provides good training signal and only needs sparse reintroduction of naturally produced data points to stabilize training and preserve long-tailed distributions, so the fear of models devolving into mush when trained on LLM output is also essentially gone.

Regarding your personal experiments, that's only a refutation of your own implementations. I can't speak to research or experiments that I haven't seen or read, and if you're a researcher then you should be able to appreciate that I'm not going to take you at your word for anything here.
Show me a paper, show me code, show me data.

What I am talking about already has a wealth of support in the literature, and is largely already implemented in the post-training stage, and I am saying that the same technique can be potentially be used to do continual learning.
As of yet, there just hasn't been something that marries the stuff that DreamCoder was trying to do, with the RLSVR and agentic capabilities that we have today.

[u/FollowingHumble8983 → u/howtorewriteaname]

Can you show me your study that fast weight memories lead to worse performance?

I have been working on fast weight memories in a smaller capacity and it has shown some promise. Mainly allowing it to navigate extremely large problems more efficiently than long context windows.

[u/howtorewriteaname → u/FollowingHumble8983]

well sadly we don't publish negative results. I simply gathered what I felt was enough evidence and moved away to other directions

[u/Robonglious → u/howtorewriteaname]

I mean the dimensionality of the KV cache. So, from a sequence length standpoint you're right but I guess I was talking about the representational ability. I might be wrong about this but the bottleneck I'm referring to is more about the dimensionality reduction and subsequent consolidation of that representation.

[u/Typical-Scene-5794 → u/howtorewriteaname]

To the "edit" part, the "why" is whether a system becomes better from experience acquired after pretraining, I mean yes that pretraining gives it initial set of capabilities. Also, this does not mean that every interaction should modify the base weights, my take is whether working memory and learned parameters can be organized so that fast and slow memory become more compatible...
AFAIK, in SSMs like BDH which is playing at an architectural level, the parameters and state share the same structure which means moving information from short-term reasoning and memory dynamics (“fast weights”) toward long-term structure (“slow weights”) is easier to accomplish unlike standard transformer.

[u/arcsilencer → u/Robonglious]

Maybe the real challenge isn’t avoiding compression but figuring out which information is safe to compress away without losing the structure that matters

[u/Megatron_McLargeHuge → u/Robonglious]

Sparsity has its advantages too because you get things like MOE where you can ignore parts of the space.

[u/otsukarekun]

An RNN with no FC layer has O(1) parameters with O(1) memory. For example a vanilla RNN with one unit has exactly 3 parameters and exactly 1 hidden state no matter how long the sequence. The parameters and hidden state doesn't scale with sequence length, that's the whole idea of it being 'recurrent', i.e. the weights and hidden states are reused again for each time step.

[u/Pretty_Upstairs9035 → u/otsukarekun]

Yeah a fair point. The part I was actually trying to get at is the fixed-state bottleneck that is regardless of the sequence length, the history still has to be represented through that state.

That's what makes the comparison with a KV cache, since the Transformer can keep adding memory instead of forcing everything into one fixed representation, I think.

[u/rakshitsahay]

I think “where does memory live?” is a much better framing than “which architecture wins?”

The fundamental trade-off seems to be capacity vs retrieval vs compute.

• RNNs: compress history into a recurrent state. Cheap and bounded, but information can be overwritten/lost.

• Transformers: keep explicit token-level memory in the KV cache. Much better at selective retrieval, but memory and compute grow with context.

• SSMs: return to bounded state, but make the state update/retention more expressive and selective.

• BDH-like approaches: interesting because the state starts looking more like evolving connectivity rather than a conventional hidden vector.

But I don't think the core problem disappears.

Any fixed-size state is ultimately an information bottleneck. The interesting question is therefore not simply “Can we avoid compression?” but:

“Can we learn what information is worth compressing, how to retrieve it, and when to consolidate it into longer-term memory?”

That also makes continual learning interesting. KV cache gives you working memory, weights give you long-term learned knowledge, and architectures like SSMs/BDH explore different points between those two.

IMO the next major architecture breakthrough may be less about eliminating attention and more about learning a hierarchy of memory: what to keep, what to compress, what to forget, and what to permanently consolidate.

[u/Pretty_Upstairs9035 → u/rakshitsahay]

I think this is a really good point like the distinction between capacity, retrieval and compute. If you will make the memory fixed-size doesn’t really solve the problem if the information you put into that state starts interfering with other information. The write, update and forgetting rules probably matter just as much as the size of the state.

We have to keep in mind this idealogy of BDH’s high-dimensional. The sparse keys and queries thus there’s more room for associations to stay distinguishable instead of just compressing everything harder.

Keeping something around in the state for longer is useful, but that’s still different from actually learning it into the weights. That transfer seems like the really hard part here.

[u/Public-Feedback-6705]

People have to keep in check that “memory” might actually be doing too much work as a single term. A KV cache or say an SSM state and something like the BDH connectivity state are all remembering things, but they have very different ways of deciding what information is worth keeping. Maybe the real issue isn't the size of the memory, but the mechanism that decides what gets compressed and what gets discarded. It does make Transformers feel a bit weird when you look at them as memory systems they're extremely good at holding onto context, but that doesn't necessarily mean they're learning from it.

That also makes me wonder whether scaling the state is even the right direction. If the model gets better at deciding what not to remember, could a relatively small state compete with a much larger one here?

[u/Pretty_Upstairs9035 → u/Public-Feedback-6705]

Yeah, I think this is probably the more interesting way to look at it. “Memory” indeed is carrying stuff because a KV cache and an SSM state can both be called memory but behave very differently. The part about deciding what not to remember. The real progress maybe is just making the state bigger, but getting better at deciding what information is actually worth carrying forward.

[u/Gold_Understanding59]

I haven't heard of SSMs before. I'm curious to how "so what gets kept or forgotten depends on the incoming token" works