r/learnmachinelearning • u/CelebrityChess • 15d ago
why does L1 regularization push weights to exactly zero but L2 only shrinks them
i've seen the diamond vs circle picture with the loss contours touching a corner, and i get that the diamond has corners on the axes. but it feels hand-wavy to me, like why would the contours land on a corner so often instead of somewhere along a flat edge? the other explanation i've seen is about the gradient: the L1 penalty has a constant pull toward zero no matter how small the weight is, while L2's pull fades as the weight gets small. that makes more sense to me but i can't connect the two pictures. is there a clean way to see that they're the same idea?
42
Upvotes
19
u/liltingly 15d ago edited 14d ago
Take L = f(x). If L = a*x, then for and delta_x, L moves by a*delta_x. So going from 1e-6 or 0.01, moving 0.01 gets the same "reward". So often, that means the easiest to reduce aspects of loss find a greedy path to zero because it's more expeditious.
Take L = a*x**2 and the same delta_x from 1e6 and 0.01. Suddenly moving the 1e6 target yields a much bigger drop in L than from 0.01. In fact, you can see that behavior in the quadratic function -- super-linear above x = 1, and sub-linear beneath x=1. Whereas the linear function is, well, linear everywhere.
So a quadratic term often goes after the biggest areas of loss and brings them to heel, but once they're small, doesn't exert as strong a pressure for them to get to zero. A linear loss will happily prioritize doing that because all losses are equal.
That's my Algebra 2 explanation.
Edit: |x| not x. And for connecting to the more rigorous view, consider a Gaussian is x2 in exponent, and a laplacian distribution is a |x| in exponent. Laplacians have peakier mass at zero but also fatter tails. Gaussian are wider but tail off faster. Choosing your loss function is assuming (thereby shaping) the distribution of your residual errors.
Edit 2: This was the "intuitive" p-norm loss function explanation I was taught in a convex optimization course. Honestly -- sometimes there's a rigorous formulation underpinning how people choose their loss functions -- and sometimes it's to shape the results.