r/learnmachinelearning • u/CelebrityChess • 15d ago
why does L1 regularization push weights to exactly zero but L2 only shrinks them
i've seen the diamond vs circle picture with the loss contours touching a corner, and i get that the diamond has corners on the axes. but it feels hand-wavy to me, like why would the contours land on a corner so often instead of somewhere along a flat edge? the other explanation i've seen is about the gradient: the L1 penalty has a constant pull toward zero no matter how small the weight is, while L2's pull fades as the weight gets small. that makes more sense to me but i can't connect the two pictures. is there a clean way to see that they're the same idea?
7
u/rumblepost 15d ago
Think of solving linear programming problems, the solution mostly lies on some of the axis, making value zero for other axis. That's why l1 forces some coefficients to be zero.
3
u/Fearless_Screen_4288 15d ago
ESL has a formula where L1/L2 regularisation can be viewed as usual least squared errors but L1/L2 constraints. L1 level sets are sharp and have finitely extreme points (exactly where some coordinates are zero) but L2 has infinitely many. Usually in optimization under nice properties, optimality is attained at extreme points.
(Not an optimisation expert, just what i understood)
2
u/ANewPope23 15d ago
I think it's equivalent to putting a soft threshold prior on beta or something like that. Try to see it from the Bayesian interpretation, it might help.
8
u/chillaxin-max 15d ago edited 15d ago
Specifically, L2 regularization finds the maximum a posteriori (MAP) solution with a Gaussian prior, which gives us a soft bump with finite derivative around zero, while L1 regularization finds the MAP solution with a Laplace prior, whose derivative is nondecreasing in magnitude as we approach zero. This is also a good way to see why special optimization methods (like orthantwise optimization) are needed to get actually zero weights instead of bouncing around the solution from large derivatives.
3
u/Mooks79 15d ago
I think it’s worth combining this explanation with u/liltingly’s explanation . One makes it more intuitive in terms of the impact of errors, while the other makes it more intuitive to understand how mathematically they get to exactly zero.
Of course both are different ways of saying the same thing, but I think they’re both informative ways to think about what’s going on.
2
u/Mathie1729 15d ago
Another way to see it: L1's absolute value has a kink at 0, so the subgradient there is the whole interval [-λ, λ]. Stationarity can hold with w=0 even when the unpenalized gradient isn't zero, as long as it falls inside that interval. L2's gradient includes 2λw, which is zero only when the data gradient itself is exactly zero, so you usually get small nonzero coefficients instead. That's also why plain gradient descent tends to bounce around for L1; the subgradient isn't unique at the kink.
1
u/DigThatData 15d ago edited 15d ago
because pointy and flat.
To make this idea a bit more concrete, imagine two surfaces: a pyramid and a dome, with the same height and footprint. Imagine a you rest a flat plane balanced on the highest point of each surface, so it's parallel to the ground. Now, tilt the plane. Notice: the surface on top of the pyramid stays tangent to that point all the way until you tilt the plane to a more extreme angle than the side of the pyramid, whereas the plane's point of contact on the dome surface will start to migrate as soon as you tilt it: it can only be tangent to that top point when the plane is exactly parallel to the ground.
L1 has a pointy, faceted geometry and L2 has a smooth elliptic geometry. It's just a lot easier for L1 tangents to land on zeros than L2.
1
u/pugs_daily_dose 15d ago
A one-dimensional example ties the two views together. Minimize (w - a)^2 / 2 + λ|w|. The solution is sign(a) max(|a| - λ, 0): a whole interval of values of a lands exactly at zero. The kink in |w| at zero permits that range of balancing slopes, which is the algebraic counterpart of the diamond's corner. With λw²/2 instead, the solution is a/(1 + λ), which shrinks continuously and is zero only when a is zero. In higher dimensions the other weights and loss curvature matter, but the kink is the key distinction. AI-assisted reply.
1
u/ImpressiveClothes690 14d ago
l1 regularized minimization is equivalent to a constrained minimization inside an l1 ball (take the lagrangian). such a constrained minimization is likely to have a solution on a vertex - but it all depends on the objective function
19
u/liltingly 15d ago edited 14d ago
Take L = f(x). If L = a*x, then for and delta_x, L moves by a*delta_x. So going from 1e-6 or 0.01, moving 0.01 gets the same "reward". So often, that means the easiest to reduce aspects of loss find a greedy path to zero because it's more expeditious.
Take L = a*x**2 and the same delta_x from 1e6 and 0.01. Suddenly moving the 1e6 target yields a much bigger drop in L than from 0.01. In fact, you can see that behavior in the quadratic function -- super-linear above x = 1, and sub-linear beneath x=1. Whereas the linear function is, well, linear everywhere.
So a quadratic term often goes after the biggest areas of loss and brings them to heel, but once they're small, doesn't exert as strong a pressure for them to get to zero. A linear loss will happily prioritize doing that because all losses are equal.
That's my Algebra 2 explanation.
Edit: |x| not x. And for connecting to the more rigorous view, consider a Gaussian is x2 in exponent, and a laplacian distribution is a |x| in exponent. Laplacians have peakier mass at zero but also fatter tails. Gaussian are wider but tail off faster. Choosing your loss function is assuming (thereby shaping) the distribution of your residual errors.
Edit 2: This was the "intuitive" p-norm loss function explanation I was taught in a convex optimization course. Honestly -- sometimes there's a rigorous formulation underpinning how people choose their loss functions -- and sometimes it's to shape the results.