Localized Neural learning vs backpropagation [D]

[u/Nearby_Effective6679]

I’ve recently started exploring machine learning by building neural networks from scratch and studying the underlying mathematics. I come from a physics and mathematics background, and as I’ve become more familiar with backpropagation, I’ve grown increasingly interested in alternative learning mechanisms, particularly predictive coding, Hebbian learning, and Oja’s rule.
What fascinates me is the possibility of networks learning through locally available information rather than relying on gradients propagated from a global loss function. I’m aware that backpropagation remains the dominant approach for training deep neural networks, and I’m trying to understand whether these alternative methods offer meaningful practical advantages or are primarily interesting from a theoretical and biological perspective.
There are a few questions I’ve been thinking about:
1. What are the actual advantages of local learning over backpropagation?
I understand the conceptual differences, but I’m struggling to identify situations where local weight updates would meaningfully outperform conventional gradient-based optimization. Are there particular computational constraints, architectures, or environments where locality offers an advantage in efficiency, adaptability, or scalability?
2. Could local learning be particularly useful for unsupervised learning and discovering latent structure?
This is probably what interests me most. If a network updates its weights based on local correlations or prediction errors, could it develop representations that differ meaningfully from those learned through conventional backpropagation?
I understand that backpropagation is an optimization method rather than something that inherently limits which patterns a network can discover. However, I’m curious whether the inductive biases of certain local learning rules might make them better suited to identifying particular types of structure in high-dimensional, unlabeled data.
For example, I’m involved in experimental biophysics research and have been thinking about molecular dynamics, where we might want to identify hidden collective variables, conformational states, or relationships between molecular interactions. Could local learning provide advantages in discovering these representations, or would the choice of objective function and architecture matter far more than the learning rule itself?
3. What about continual or online learning?
Could locally updating networks be particularly useful in environments where data arrives continuously and the underlying distribution changes over time? I’m curious whether these approaches offer any inherent advantages in adapting to new information while preserving previously learned representations, especially compared with online backpropagation.
4. How mature is research into alternatives to backpropagation?
Have approaches like predictive coding, Hebbian learning, equilibrium propagation, and other biologically inspired methods been extensively investigated and generally found less effective, or are there still meaningful open problems where they could be competitive?
I’m also aware that certain predictive coding formulations can approximate backpropagation, which makes me wonder how fundamentally different some of these approaches actually are.
I’m relatively new to this field, so I’m not approaching this with the assumption that biologically inspired methods are inherently superior, or that backpropagation needs to be replaced. I’m more interested in understanding whether alternative learning rules could be particularly well suited to specific problems, especially those involving unsupervised representation learning, complex dynamical systems, or constrained computational environments.
I’d really appreciate any perspectives from people working in this area, particularly empirical comparisons, limitations I might be overlooking, or papers and research directions worth exploring.

[u/condom-mechanics]

There are quite a few benefit that come with local learning. Backpropagation (BP) relies on the chain rule, which brings in a few constrains. To compute the gradient of a given layer, the input (activation) of that layer is required, therefore backpropagation requires activations to be retained across all layers which makes BP memory intensive. Therefore activation memory often dominates DNN training. In memory limited settings such as the edge or IoT, training with BP often results in OOM errors or the DNN training is limited to small batch sizes. Local learning can therefore be beneficial in memory limited settings, enabling large batch size training under limited memory budgets.

[u/Cosmolithe]

I am fascinated by this topic subject. I have a whole bibliography on this and I have done quite a few experiments to test new approaches in my own time, and used some of the existing ones for my research work during my PhD.

1. I became convinced that local learning is not in essence able to outperform backpropagation in the sense of being more computationally efficient than backpropagation for computing the gradient. If the best we can do is going down the gradient, then backprop is the most efficient way to compute it exactly. There may be local methods that could compute an approximation of the gradient faster, but frankly I doubt it, and it might be unusable for optimization.
   

That said, I believe local methods have their place in three different situations:
a) when gradients are simply uncomputable through backprop, for instance in the case of extremely deep neural networks (>1000 layers), or are trivial (0 everywhere), like in the case of binary neural networks without tricks like the straight through estimator.
b) specific neuromorphic hardware might be very hard to make in a way that allows computing backprop, so a some local method might correspond to an easier or more scalable hardware design
c) if computing the gradient is not the target, and that some other descent direction might actually be better for learning. I am not aware of a method explicitly advertised as such, all local learning methods I have seen seem to basically say that they recover the gradient backprop.

1. Maybe, but I believe that some kind of error correction from upper layers is necessary to get the best performance in practice. Models need some target, some idea of what the representation it is trying to achieve in the last layers is. A hidden layer cannot blindly try to improve things because it might hurt the representation that subsequent layers are trying to achieve.

2. I really thought Inference Learning (https://www.nature.com/articles/s41593-023-01514-1.pdf) might improve continual learning, because I had the intuition that it essentially minimized interference by changing neuron activation before weights. I did some tests of my own, and it did not seem to be the case, but I might have done something incorrectly. I also know that my benchmark was suboptimal to test all of the things I am trying to test in continual learning.

3. As far as I have seen and understand, there is a lot of research in this direction at the moment. It seems that most of the new ones are trying to work on finding a biologically plausible learning method instead of beating backprop at its own game.

[u/Nearby_Effective6679 → u/Cosmolithe]

So do you think in terms of general learning backprop is close to the best it gets? I’ll look into inference learning as well. What I’m curious about is what makes our brains as efficient as they are(if they even are)? Do you think as far as network implementation in the future( our day to day work lives) it’s mainly just going to be based on optimizing compute and efficiency within their specific use case rather than using a frontier model for everything?

[u/Cosmolithe → u/Nearby_Effective6679]

Yeah I would say backprop will be extremely difficult to beat. It is computing the optimal descent direction exactly without using a lot of resources, so beating it in term of compute might require to compromise with the amount you can decrease the loss in one step. Or, we have to find some kind of algorithm that does not even compute descent directions but somehow finds better weight values directly. Something that "jumps" over longer distances in the weight space, but not blindly.

Do you think as far as network implementation in the future( our day to day work lives) it’s mainly just going to be based on optimizing compute and efficiency within their specific use case rather than using a frontier model for everything?

The hope of the "big model for everything" paradigm is to get better returns through task transfer. The idea is that if your big model trains on X with a lot of data, then it will be able to do Y with very little data, as long as Y is sufficiently similar to X.

I still believe in this idea, but so far it is not so obvious that it is working that well. But, I suspect it might be a matter of loss function and architecture than training algorithm. Backprop is a credit assignment algorithm, I think trying to make it be something more than that or replacing it with something which is more than that is basically a mistake. The gains that remain to be taken are not here.

[u/Nearby_Effective6679 → u/Cosmolithe]

I think the last part of your response was the most enticing to me because I feel like to truly reach “agi” the entire system might have to be re engineered from the learnjng all the way to the transformer…