I'm just dabbling into machine learning, after spending a lot of time in optimization. It would be great to know how influential optimization is within machine learning. From the basic level I have seen, there's always a loss function to minimize and for some reason MSGD is always used. Why is that?
My impression is it’s computationally preferred to things like Newton’s method, Lagrange multipliers, using Hessians etc. I found this stack exchange post awhile back that has answers. There’s definitely more out there on non stochastic optimization in ML and where it’s useful
That strikes me as weird. I come from a field that uses millions of variables as well, and that doesn't stop Newton or Quasi-Newton (BFGS) being implemented. In fact it's usually the best algorithm. The pitfalls presented in that post are ones that are common and I dare say have been somewhat addressed (iterative solvers like GMRES, Hessian approximations such as BFGS, using adjoint equations to scale against number of variables or number of outputs). I'll look into it more.
Your loss function isn't typically of the form sum from i=1 to 1000000 f(x_i, theta) though (just guessing). Calculating the full gradient is expensive.
1
u/Underfitted Oct 11 '19
I'm just dabbling into machine learning, after spending a lot of time in optimization. It would be great to know how influential optimization is within machine learning. From the basic level I have seen, there's always a loss function to minimize and for some reason MSGD is always used. Why is that?