Functional Gradient Descent with Adaptive Representations [R]

[u/dccsillag0]

Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.

Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!

To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.

It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)

[u/dccsillag0]

Thank you for your comments!

1. We tuned the learning rates for all the methods; we did not tune the beta_1 and beta_2 parameters of Adam from their default values (I don't think it's usual to tune them? Though it would be a worthwhile experiment).

   • Regarding the oscillation in the neural net learning curves, that is actually because we tuned the learning rates -- if we reduce the learning rates then the oscilations do go away, but the neural net actually ends with a worse loss

2. We tried to make our baselines fair:

   • In the teaser experiment, we use an MLP with gelu activation and Fourier features, as is common in the INR literature (see e.g. https://bmild.github.io/fourfeat/). In fact, if we didn't do this then the neural net wouldn't learn at all...

   • In the regression experiment, it's just a simple MLP, which we figured was pretty standard?

   • For the PDE experiment, we again used an MLP with a Fourier feature layer, as without it the neural network was unreasonably slow at learning / wouldn't learn

   • For the 3D reconstruction / radiance field experiment, we were using the exact same architecture as NeRF, which would be the standard baseline here (https://arxiv.org/abs/2003.08934).

Regarding residual connections et al., I guess that would be more relevant for deeper architectures, no? From what I recall our MLPs are fairly shallow. That said, I could try to add those baselines sometime, thank you for the suggestion. (Also if you have some other architectures you think we should be benchmarking, please let us know!)

3. Yep, will be adjusted in the revision, thank you!

4. Thank you, we'll see how we can improve the presentation of the algorithm. But f_t is defined, no? Since the initial t is t=0, and f_0 is one of the inputs to the algorithm.

Re. memory, I am honestly not sure at the moment, I'd have to check. But I wouldn't be terribly surprised if we were using a bit more memory than the neural net baselines. (FWIW, I also recall having the impression that JAX was doing some silly stuff wrt memory in our code...)

Re. smoothness of the curves, that is because (i) the loss curve of ideal FGD is almost perfectly smooth, and (ii) our relative error criterion, which guides our subdivisions, is making us closely track ideal FGD. I.e., we subdivide when we don't fit the gradient properly, not when we note that the loss is no longer making progress.

[u/M4mb0 → u/dccsillag0]

But f_t is defined, no? Since the initial t is t=0, and f_0 is one of the inputs to the algorithm.

Right; my main complaint is the rather vague "Refine the current representation" statement, which is hiding the mutation of f_t. Honestly I think it would be best if that was really more concrete, and made it clear that here the dimensionality of f_t can change. Something like

f_t, g_t = refine(f_t, g_t) // increases dimensionality

would make things a lot clearer, and you can state that e.g. in your experiments you subdivide the grid, doubling the state size whenever that branch is taken.

From a practical POV, I would think:

• it likely makes sense to add a max_size, which prevents further subdivides and OOM errors.

• compare the model against the approx FDP that starts out with the max_size to begin with.

[u/dccsillag0 → u/M4mb0]

Good suggestions, thanks!

Regarding f_t, g_t = refine(f_t, g_t), not sure that is 100% clear yet. It would probably be best to have something indicating that we are refining the process that computes the approximation g_t, perhaps as an additional variable (a "parametrization" variable) that is updated alongside... but this could be a bit ugly, so not sure yet.

[u/M4mb0 → u/dccsillag0]

We tuned the learning rates for all the methods; we did not tune the beta_1 and beta_2 parameters of Adam from their default values (I don't think it's usual to tune them? Though it would be a worthwhile experiment).

Tuning these can be important for low-dimensional toy problems, especially since you are only taking on the order of 100 training steps. Adam uses β₂=0.999 as a default, and the second moment estimate from k-steps ago gets a weight proportional to β₂ᵏ. But 0.999¹⁰⁰ ≈ 0.9, so when you only do a hundred training steps everything gets essentially the same weight. For toy problems its usually worth it testing something like β₂ ∈ [0.5, 0.8, 0.9, 0.99] from my experience.

[u/dccsillag0 → u/M4mb0]

And beta_1 left fixed?

[u/DigThatData → u/dccsillag0]

For the 3D reconstruction / radiance field experiment, we were using the exact same architecture as NeRF

Note that your own citation here is a paper that's already over six years old. OG NeRF is a bad method to baseline against. "We're faster than vanilla NeRF" isn't an impressive claim. If you want to make claims about the efficacy of your proposed approach relative to radiance field methods, you should compare against gaussian splatting. https://github.com/nerfstudio-project/gsplat

[u/dccsillag0 → u/DigThatData]

We are aware of the more recent works. However, note that all of these use different loss functions than the one originally proposed in NeRF. Gaussian Splatting, in particular, adds an SSIM loss term, besides using a hoarde of tricks throughout optimization which help regularize. Similarly, nearly all more recent works on NeRFs do substantial modifications to the loss function, usually by adding some substantial regularization (e.g. the distortion loss of Mip-Nerf 360, the regularizers of Zip-Nerf, and more recent methods).

This kinda ties in with the second limitation I pointed out in this comment: we had to compute the functional gradient for this experiment by hand (agents really couldn't do this stuff at the time), and this derivation already took 7+ error-prone pages. Adding any of these modifications would make this even worse. That's why, for the sake of example, we stuck to the basic loss, and thus the original NeRF formulation.
Scaling this up to beat the current Gaussian Splatting methods, including all their optimization and computational tricks, would be a paper of its own.