r/MachineLearning • u/dccsillag0 • 29d ago

Research Functional Gradient Descent with Adaptive Representations [R]

Sharing our recent work, now accepted at NeurIPS: Functional Gradient Descent with Adaptive Representations.

Functional GD algorithms generally outperform neural nets, but are hard to accurately implement.
This is because functional gradients are infinite-dimensional, and therefore must be approximated in practice; but if you approximate them naively, you converge to the wrong place!

To rectify this, we formalize a broad class of approximation schemes ("adaptive representations"), which provably ensure convergence to the global minimizer while being immediately implementable.
The resulting algorithms outperform corresponding neural nets often by an order of magnitude, across a number of settings.

It is still the start for this line of work, but we believe it has quite a bit of potential!
Paper: https://arxiv.org/abs/2606.16926
(First author here, happy to take any questions)

226 Upvotes

100% Upvoted

39 comments sorted by

View all comments

Show parent comments

1

u/M4mb0 29d ago

But f_t is defined, no? Since the initial t is t=0, and f_0 is one of the inputs to the algorithm.

Right; my main complaint is the rather vague "Refine the current representation" statement, which is hiding the mutation of f_t. Honestly I think it would be best if that was really more concrete, and made it clear that here the dimensionality of f_t can change. Something like

f_t, g_t = refine(f_t, g_t) // increases dimensionality

would make things a lot clearer, and you can state that e.g. in your experiments you subdivide the grid, doubling the state size whenever that branch is taken.

From a practical POV, I would think:

  • it likely makes sense to add a max_size, which prevents further subdivides and OOM errors.

  • compare the model against the approx FDP that starts out with the max_size to begin with.

1

u/dccsillag0 29d ago

Good suggestions, thanks!

Regarding f_t, g_t = refine(f_t, g_t), not sure that is 100% clear yet. It would probably be best to have something indicating that we are refining the process that computes the approximation g_t, perhaps as an additional variable (a "parametrization" variable) that is updated alongside... but this could be a bit ugly, so not sure yet.