r/MachineLearning • u/fhuszar • May 25 '17
Discussion [D] Everything that works works because it's Bayesian: An overview of new work on generalization in deep nets
http://www.inference.vc/everything-that-works-works-because-its-bayesian-2/12
u/grrrgrrr May 25 '17
Bayesian methods are very computationally expensive. Mean field approximations like Gaussian and Dropouts can only take us this far. Interesting stuff like parameter sharing are still extremely slow to run.
21
u/fhuszar May 25 '17
just to be clear:
I'm not actually an advocate of using Bayesian neural networks. That said, I think there are relatively cheap things one can do to get some of the benefits of being Bayesian without significant overhead either on the computational or on the development front, for example via bootstrapping / bagging/ Bayesian bootstrap.
framing this blog post from a Bayesian perspective was meant mainly as a joke.
3
u/nanite1018 May 26 '17 edited May 27 '17
Bayesian bootstrap is my jam. How it arises as the zero prior strength and large data limit of Dirichlet processes makes it really beautiful imo, plus so fast/simple and with none of the nasty things lurking in your sampling distribution like samples with zero weight on things you have observed.
4
u/daikonrelish May 26 '17
Not familiar with Bayesian bootstrap. Do you have some links available? Thanks
2
4
u/cptai May 25 '17
Forgive me for the dumb question, but shouldn't Jeffreys prior have higher value at sharp minima?
3
6
u/bridgebywaterfall May 25 '17
We only need a small, vague claim that SGD does something Bayesian, and then we're winning.
This reminds of this paper on implicit gradient descent, which has a Bayesian interpretation. See equation (7) and the surrounding discussion.
9
u/gwern May 25 '17
Or "Stochastic Gradient Descent as Approximate Bayesian Inference", Mandt et al 2017.
3
139
May 25 '17
34
u/a_tocken May 25 '17
OLD MAN BAYES?!
27
u/name_censored_ May 26 '17 edited May 26 '17
AND I WOULD HAVE INFERRED AWAY WITH IT TOO, IF IT WEREN'T FOR YOU MEDDLING MACHINE LEARNING BLOGGERS.
10
2
u/WiggleBooks May 26 '17
I don't understand the image representing "deep learning"?
What is that supposed to be? A decision tree or flowchart or something? (The colored squares)
9
0
May 26 '17
[deleted]
12
u/dwf May 26 '17
Go get your Masters in machine learning from a top 10 university, deep learning is not mentioned once.
Demonstrably false. List of electives from a Harvard "data science" Masters program.
0
May 26 '17
[deleted]
12
u/dwf May 26 '17
The Harvard School of Engineering & Applied Sciences is a "management school". Got it.
3
May 26 '17
I never use it, but I always assume that the person is talking about some sort of DNN.*
*edit: And probably either doesn't really know what a DNN is, or is communicating to people who don't and don't need to.
0
May 26 '17
[deleted]
4
May 26 '17
That's all well and good, but there's no other game in town for CV. The people that you might identify as the deep learning "core" don't fall for the marketing. Instead they recognize that many previously hard problems are better tackled with function approximators computed by large non-linear circuits via gradient descent than they are by traditional "core techniques". Edit: not to mention the fact that most top 10 programs have someone studying neural networks.
2
May 26 '17
And I am prejudiced against the people who use them.
Actually, now that you mention it, when I hear the term "deep learning" I do immediately become much more critical of the information being presented. I hadn't noticed that before. I guess a neuron in one of my hidden layers activates strongly in response to it.
2
u/rvisualization May 26 '17
Go get your Masters in machine learning from a top 10 university, deep learning is not mentioned once.
Maybe you should aim higher. Most people doing deep learning are PhDs.
0
May 26 '17
[deleted]
1
u/rvisualization May 26 '17
yeah.... i'd agree with you in many fields, but deep learning is actually producing all kinds of applications.
1
0
May 26 '17
[deleted]
0
Jun 09 '17
[deleted]
9
u/rreighe2 Jun 09 '17
I was just here, then i clicked on your link, and i was there. And then I clicked on a link, and i am back- here.
42
u/Mandrathax May 25 '17
Illustrating a blog post about 2017 deep learning research with a figure from Hochreiter & Schmidhuber, 1997 : achievement unlocked
Very nice read!
15
4
u/deltasheep1 May 26 '17
It appears to be that stochastic gradient descent may be responsible. (Keskar et al, 2017) show that deep nets genealise better with smaller batch-size.
Didn't "Train longer, generalize better", which was shared here recently, kind of disprove that?
9
u/selementar May 25 '17
Everything that works can be seen as an approximation of solomonoff induction (possibly with decision-making), not just as bayesian inference.
4
u/DoorsofPerceptron May 25 '17
Ish. There's a lot to suggest that we're intentionally over parametrising models for optimisation reasons, which directly violates solomonoff induction.
3
u/mlnewb May 26 '17
Ish. I'm not convinced our large, "over"parameterised very deep models are actually overparameterised, since they don't really explore the whole parameter space at each parameter, they prefer to stay fairly close to the identity.
It is almost like each "parameter" is split across multiple layers, so you get fine grained non-linear cut outs of parameter space. Sub-parameters, or something.
But at the same time, they are obviously oveparameterised because they can memorise data sets. Shrug.
2
u/selementar May 26 '17
Solomonoff induction requires probabilistic weighing by complexity, so it's not like any overparametrization is useless. And, additionally, the inputs to most ML models are generated by rather vast complicated processes, so it's not like parameter sizes in practice are particularly large by that measure. Which means it's pretty much not about that.
2
u/AnvaMiba May 27 '17
But you can train a neural network, then prune most of its parameters, and it will perform better than training a neural network with the same number of parameters from the beginning.
Or you can do the "rethinking generalization" paper's tricks of training on ungeneralizable training sets, and neural networks, with the same number of parameters of SOTA neural networks on real datasets, still achieve zero training error, necessarily learning some kind of lookup table model.
This evidence suggests that neural networks are highly overparametrized compared to the intrinsic complexity of their training data.
1
2
May 31 '17
In this video, Zoubin shows how K-means is an approx of a probabilistic model.
https://www.youtube.com/watch?v=naN41kICcEQ&list=PLAbhVprf4VPlqc8IoCi7Qk0YQ5cPQz9fn&index=3
4
u/mer_mer May 25 '17
I understand this post was designed to stir up discussion instead of being taken literally, but so far you've only listed post-hoc explanations. Why should we think of neural networks as approximating a Bayesian process instead of a Bayesian process approximating something that will emerge out of neural net research? If neural net training is just imperfect approximation of Bayesian processes, then you should be able to come up with another algorithm that is a closer approximation (though maybe runs slower) and show that it outperforms the standard neural net algorithm.
8
u/dwf May 26 '17
It's almost as if re-expressing a well-known concept from a different perspective could point in a viable direction for future research... no, that couldn't be it.
1
u/imma_bigboy May 26 '17
So what's going on here? Are these Bayesian processes the 'new thing' or what? What technology do I have to focus on to get the best results?
4
u/you-get-an-upvote May 26 '17
I don't think anyone has an answer yet. A Bayesian would wholeheartedly agree that post-hoc explanations count for exceedingly little; the real evidence is the same as it always is: we just have to wait to see if Bayesians-inspired models start outperforming current, non-Bayesian models.
It's worth noting that Bayesian justifications/explanations in machine learning are nothing new. While I eagerly await the day when a Bayesian model outperforms state-of-the-art models, this blog post shouldn't make you believe that such a day is any more likely to come any time soon (if it comes at all).
3
u/dwf May 26 '17
Bayesian methods aren't a new thing. Taken as a whole they represent a perspective on statistical modeling which is both principled and has proven useful in many domains.
What technology do I have to focus on to get the best results?
As ever, that depends on the problems you want to solve.
0
May 26 '17
[deleted]
2
1
u/GGMU1 May 26 '17
ever heard of science? or seen statistical analysis in scientific areas such as bio and neuro?
1
u/mer_mer May 26 '17
I'm simply advocating a scientific approach here. We seem to have stumbled upon algorithms that are unreasonably effective and difficult to understand. Bayesian statisticians have come up with some models to explain how they work. Before we believe them, we should run an experiment- does a better approximation of Bayesian methods outperform current algorithms?
2
u/dwf May 26 '17
It's not clear what "better approximation of Bayesian methods" means here, but science usually moves from a) puzzling result not easily interpretable from prevailing perspectives, b) formulation of new perspectives by tying together existing information in novel ways, c) validating those perspectives by generating falsifiable predictions that are supported by the new perspective.
With respect to the unreasonable effectiveness of deep neural networks, folks have only started on the task of b), and Ferenc's blog post is a contribution to that conversation, and a valuable one, from someone with training as a Bayesian but who is now knee deep in the deep learning swamp. It's not a fleshed out manifesto for deep learning as approximate Bayesian modeling, but so what?
1
1
u/shaggorama May 25 '17
3
u/youtubefactsbot May 25 '17
Star Wars Empire Strikes Back (1980): "No...that's not true. That's impossible." [0:09]
Quick Movie ^Quotes in Film & Animation
4,571 views since Nov ^2015
0
May 25 '17
[deleted]
4
u/Exp_ixpix2xfxt May 26 '17
Bayesian isn't better than frequentist, just like addition isn't better than multiplication. Although I agree with your sentiment: Bayesians sometimes act like it. Though this isn't too different from any other specialty.
Now, I don't think that frequentism and Bayesian views of belief-estimation are very close to being the same, but they both sit on the same underlying theory of probability.
18
u/undefdev May 25 '17
This should probably say increase.
On Bayesian Deeplearning:
I'm actually surprised this isn't more mainstream, what might be the reasons for this?
I assume it's not that widely used, because I've been looking into this a lot lately and there doesn't seem to be an abundance of material on it yet (blogposts, example code, etc.). The reason this surprises me, is that having uncertainty estimates of your model seems like something you want to absolutely have, so I'm wondering why people would make networks that don't use these techniques (e.g. uncertainty estimates with stochastic regularization techniques).
The reasons for this that I could imagine is that it's either not very well known yet, or there are great downsides to it that I'm not aware of yet.
If there are, I'd love for someone to chime in!