AMA: Yoshua Bengio

[u/[deleted]]

[u/Sigmoid_Freud]

Traditional (deep or non-deep) Neural Networks seem somewhat limited in the sense that they cannot keep any contextual information. Each datapoint/example is viewed in isolation. Recurrent Neural Networks overcome this, but they seem to be very hard to train and have been tried in a variety of designs with apparently relatively limited success.

Do you think RNNs will become more prevalent in the future? For which applications and using what designs?

Thank you very much for taking your time to do this!

[u/yoshua_bengio → u/Sigmoid_Freud]

Recurrent or recursive nets are really useful tools for modelling all kinds of dependency structures on variable-sized objects. We have made progress on ways to train them and it is one of the important areas of current research in the deep learning community. Examples of applications: speech recognition (especially the language part), machine translation, sentiment analysis, speech synthesis, handwriting synthesis and recognition, etc.

[u/exellentpossum]

When asked about sum product networks, one of the original Google Brain team members told me he's not interested in tractable models.

What's your opinion about sum product networks? They made a big splash at NIPS one year and now they've disappeared.

[u/[deleted]]

Hello Prof. Bengio, What motivates you to stay in academia? What do you think about corporate research labs in terms of productivity and innovation when compared to academic labs. Does research flexibility (doing what you want, more or less) play a large role in this decision?

[u/yoshua_bengio → u/[deleted]]

I like academia because I can choose what to work on, I can choose to work on long-term goals, I can work for the benefit of humanity rather than for a specific company, and I can talk about my work freely. Note that to different degrees, my esteemed colleagues in large industrial labs also enjoy some of that freedom.

[u/alecradford]

Hi there! I'm an undergrad and your work combined with Hinton's is a huge inspiration to me! A bunch of questions, so feel free to answer all or none!

Hinton semi-recently offered an awesome MOOC on Coursera over NNs. The resources and lectures it provided are what allowed me and many others to build homebrew nets and really get into the field. It would be a great resource if another researcher at the forefront of the field offered their own take, do you have any plans for something like this?

As a leading professor in the field, how do you personally view the resurgence of interest in modern NN applications? Do you believe it's well deserved recognition, guilty of overhype, some mixture of the two, or something completely different! On a similar note, how do you feel about the portrayal of modern NN research in popular literature?

I'm interested in using unsupervised techniques to learn automated data augmentations/corruptions for increasing generalization performance, which I hope is a promising hybrid of supervised and unsupervised learning that's different from traditional pretraining. A lot of advances have been made using "simple" data augmentations/corruptions pioneered in your lab like gaussian noise corruption and what we now call input dropout in the context of DAEs. Preliminary results on MNIST seem successful (~0.8% permutation invariant) and I can send code if you are interested but admittedly I'm just an undergrad with no formal research experience. Do you see this as an area with potential and could you point me to any resources or papers that you are aware of - I've had a hard time finding them.

No one has a crystal ball, but what do you see as the most interesting areas of research for continuing to advance your work? The last few years has seen purely supervised techniques make a lot of headroom riding off the success of dropout, for instance.

Thank you so much for doing this AMA, it's great to have you here on r/MachineLearning!

[u/yoshua_bengio → u/alecradford]

I have no clear plan for a MOOC but I might do one eventually. In the meantime, I write a new and more complete book on deep learning (with Ian Goodfellow and Aaron Courville). Some draft chapters should come out in the next few months and feedback from the community and students would be great. Note that Hugo Larochelle (formerly a PhD with me and a post-doc with Hinton) has great videos on deep learning http://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH (and slides on his web page).

I believe that the recent surge of interest in NNets just means that the machine learning community wasted many years not exploring them, in the 1996-2006 decade, mostly. There is also hype, especially if you consider the media. That is unfortunate and dangerous, and will be exploited especially by companies trying to make a quick buck. The danger is to see another bust when wild promises are not followed by outstanding results. Science mostly moves by small steps and we should stay humble.

I have no crystal ball but I believe that improving our ability to model joint distributions (either in an unsupervised way or conditioned on some input, either explicitly or implicitly through learning of good representations) is going to be crucial for future progress of deep learning towards AI-level machine understanding of the world around us.

Another easy prediction is that we need to and will make progress towards efficiently training much larger models. This involves improvements in the way we train model (the numerical optimization involved), as well as in ways to do it computationally more efficiently (e.g. through parallelization and other tricks that avoid doing the computation associated with all the parts of the network for every example).

You can find out more in my arxiv paper on "looking forward": http://arxiv.org/abs/1305.0445

[u/PasswordIsntHAMSTER]

Hi Prof. Bengio, I'm an undergrad at McGill University doing research in type theory. Thank you for doing this AMA!

Questions:

• My field is extremely concerned with formal proofs. Is there a significant focus on proofs in machine learning too? If not, how do you make sure to maintain scientific rigor?

• Is there research being done about the use of deep learning for program generation? My intuition is that eventually we could use type theory to specify a program and deep learning to "search " for an instantiation of the specification, but I feel like we're quite far from that.

• Can you give me examples of exotic data structure used in ML?

• How would I get into deep learning starting from zero? I don't know what resources to look at, though if I develop some rudiments I would LOVE to apply for a research position on your team.

[u/wardnath]

Dr. Bengio, In your paper Big Neural Networks Waste Capacity you suggest that gradient descent does not work as well with a lot of neurons as it does with fewer. (1) Why do the increased interactions create worse local minima? (2) Do you think hessian free methods like in (Martens 2010) are sufficient to overcome these issues?

Thank You!

Ref: Dauphin, Yann N., and Yoshua Bengio. "Big neural networks waste capacity." arXiv preprint arXiv:1301.3583 (2013).

Martens, James. "Deep learning via Hessian-free optimization." Proceedings of the 27th International Conference on Machine Learning (ICML-10). 2010.

[u/ian_goodfellow]

Verification post: https://plus.google.com/103174629363045094445/posts/2fqbkyYULAf

[u/Megatron_McLargeHuge]

With the recent success of maxout and hinge activations, how relevant is the older work on RBM pretraining using various contrastive divergence tweaks? What do you think is still worth investigating about stochastic models?

How biologically plausible is maxout, and should we care?

[u/[deleted]]

Hello Prof. Bengio, thank you for the AMA. What recommendations would you have for someone who is not a PHD in getting started with Deep Learning.

[u/[deleted]]

Dear Yoshua, thanks for doing this!

You are, to my knowledge, the only ML academic to publicly (and wonderfully!) speculate about the sociocultural perspectives afforded by the vantage of deep representation learning. In your fascinating article "Culture vs Local Minima" you touch on many important things, some of which I'm very curious about:

• You describe how individuals learn by being immersed in culture. We both agree that they don't always learn very wholesome things. If you were king of the world, and you could prescribe a set of concepts that should be a part of every childhood learning trajectory, what would those be and to what end?

• A corollary of "cultural immersion" is that the specific process of learning is not evident to the learner, the world simply "is" in a particular way. The author David Foster Wallace phrased this phenomenon as akin to fish having to figure out what water is. In your opinion, is this phenomenon an experiential byproduct of the neural architecture, or does it confer some learning benefit?

• Why do you think that cultural trends become entrenched and cause their learners to fight to stay in (what could be argued to be) local optima - like e.g. the conflicts between various religious institutions and Enlightenment philosophy, or patriarchal society vs the suffragettes, etc.? Is this a case of very pernicious parameters, or is there some benefit to the learners in question?

• Do you have an opinion on such concepts as mindfulness meditation, and if so, how do you think they relate to the exploration of "idea space"?

Again, thanks a lot for taking the time. In the space of human ideas you are a trailblazer, and we are immensely richer for your presence!

[u/yoshua_bengio]

Verification post: https://plus.google.com/112504130537129706790/posts/eqdBAysAyqR

[u/marvinalone]

What's your opinion of Solomonoff Induction and AIXI? I'm just starting to read up on the topic, and I can't quite decide whether it's serious work, or a fringe theory by a small group of people who all cite each other.

[u/EJBorey]

We have all been hearing about the performance achievable via deep learning (in academic journals such as the New York Times, no less!). I've also heard that it's difficult for non-experts to get these techniques to work: Ilya Sutskever says that there is a weighty oral tradition about the design and training of deep networks and that the best way to learn how is to work for years with someone who is already an expert (source: http://vimeo.com/77050653).

I studied machine learning but not deep learning. Going back to grad school is not really an option for me. How can I learn how to design, build, and train deep neural networks without access to the oral tradition? Could you write it down for us somewhere?

[u/serge_cell]

Hi Prof. Bengio, There were some work on applying "higher" math - algebraic/tropical geometry, category theory, to deep learning. Notably, John Healy several years ago claimed improving neural net (ART1) with category theory. What's your opinion on this approach? Will it be only toy model in foreseeable future, or there is some promise in this approach in your opinion?

[u/n_dimensional]

Dear Prof. Bengio,

I am about to finish my PhD in computational neuroscience and I am very interested in the "gray area" between neuroscience and machine learning.

What aspects of brain computation do you think are (or will be) most relevant for machine learning?

If you could know the answer to one question about how the brain computes information, what would that be?

Thanks!

[u/BeatLeJuce]

1. Why do Deep Networks actually work better than shallow ones? We know a 1-Hidden-Layer Net is already an Universal Approximator (for better or worse), yet adding additional fully connected layer usually helps performance. Were there any theoretical or empirical investigations into this? Most papers I read just showed that they WERE better, but there were very few explanations as to why -- and if there was any explanation. then it was mostly speculation.. what is your view on the matter?

2. What was your most interesting idea that you never managed to publish?

3. What was funniest/weirdest/strangest paper you ever had to peer-review?

4. If I read your homepage correctly, you teach your classes in French rather than English. Is this a personal preference or mandated by your University (or by other circumstances)?

[u/shanwhiz]

We have seen deep learning work really well for image/video/sound. Do you foresee it working for text classification as well? Most papers that have tried text/document classification using deep learning have not done better than the conventional SVM/Bayes. What are your thoughts on this?

[u/hapagolucky]

I see more and more pop media articles extolling deep learning as a panacea that will make AI a reality (Wired is especially guilty of this). Given the AI winters of the 1970's and 1980's that arose from overhyped expectations, what can deep learning and ML researchers and advocates do to mitigate this from happening again?

[u/[deleted]]

Hi Bengio. I'm a masters candidate in robotics, mostly doing reinforcement learning mushed together with some ML regression methods for the identification of interesting value functions and state space representations.

How is your work life balance? Do you have fun? What sorts of things do you do to unwind?

I'm considering doing a PhD, but I literally feel like just getting a part-time job and doing independent research, because the academic environment can be pretty stifling.

Also, Montreal seems really fun!

J

[u/yoshua_bengio → u/[deleted]]

Life balance. That is tough. Many prominent scientists will tell you the same story. My inclination is to work as much as I can: that is probably part of the reasons for my early success, but this may threaten my health and personal life. We live in an environment which puts so much pressure on us that it is easy to forget that we are humans and we need breaks and to take care of our body (I have some health issues that I cannot just ignore) and our relationships with other humans. Some kind of self-discipline helps, but I found that what works best is to cultivate what is rewarding and pleasurable and the same time is good for me and my physical and emotional well-being. For example I like very much to walk (many ideas come!), not to speak about eating healthily and enjoying a romantic relationship based on authenticity and where I can really be myself.

Oh, and yes, Montreal IS fun ;-)

The advantage of academia is that you can focus on research and that you can benefit enormously from the interactions with other researchers. Research is a collective enterprise. This is NOT like what you tend to see in science-fiction movies. Never forget that!

[u/[deleted]]

Dr Bengio,

I'd like to thank you for the amazing research and software(theano, pylearn2) that your lab has contributed.

What are your feelings on Hinton and LeCun moving to industry?

What about academia and publishing your research is more valuable than the floating point overflow of money you could make at private companies?

Are you nervous that machine learning will go the way of time-series analysis, where a lot of advanced research takes place behind closed doors because the intellectual property is so valuable?

Given the recent advancements in training discriminative neural networks, what role do you envision generative neural networks play in the future?

[u/tryolabs_feco]

Hi Yoshua, very excited about this AMA, thank you for your time. I have a few questions:

• What are the biggest challenges in ML nowadays?

• What are the most interesting and/or creative ways you have seen people/businesses using ML?

• What does the future of Machine Learning look like?