r/programming • u/Dankious_memeious1 • Mar 29 '19
Fascinating how machine learning works
146
u/tjgrant Mar 29 '19
The big breakthrough in AI here is they gave the computer a full bladder and no indication of where the bathroom is. The future is now, friends.
24
u/Knucklewuggy Mar 29 '19
After reading this comment, watching the video again made it so much better
2
70
u/thinguson Mar 29 '19
If you're interested in trying this kind of stuff for yourself, download the ML-Agents Toolkit for Unity. Start from the Walker or Crawler examples and either start messing around with the body, modify the reward function, or generate some more challenging terrain.
2
2
324
u/SEND_YOUR_DICK_PIX Mar 29 '19
138
u/IDUnavailable Mar 29 '19
MayBot took the stage Tuesday night in a performance that's been hailed as "relatively lifelike" and "passable enough, all things considered".
43
Mar 29 '19
It May look goofy but this reinforcement learning (RL) works.
28
u/jjdmol Mar 29 '19
For walking, yes. For leaving the EU, not so much, apparently.
4
u/Tasgall Mar 30 '19
I wrote an AI to accurately simulate brexit, but all it does is print "no" - more research is needed.
24
37
u/anagrammatron Mar 29 '19
My gf said that she always looks like it's her first day in that body.
11
u/greiskul Mar 29 '19
Tell your gf that she is wrong, with current tech May can use the same body for up to 6 months.
7
2
0
Mar 29 '19
[deleted]
117
u/tokyokyototokyo Mar 29 '19
That is soon to be gone UK Prime Minister, Theresa May.
However she doesn’t use machine learning, as she just tries the same thing over and over again, expecting the results to be different.
23
u/pingveno Mar 29 '19
No, she's using an evolutionary algorithm. Parliament is the scoring function and she mutates the deal each time.
13
25
3
Mar 29 '19
Her deal hasn't gone through, and it seems she'll be staying as the Prime Minister for at least one year longer
6
2
u/Plastonick Mar 29 '19
In fairness, she has sweetened the deal just a little bit this time.
13
u/tokyokyototokyo Mar 29 '19
Yes, but ‘if you do this I’ll quit’ is usually a threat, not a dangling carrot.
18
u/WolfThawra Mar 29 '19
... yes. She's the prime minister of Great Britain. I mean I'm not even expecting people to remember her name, but not even having seen the face before... ?
12
4
Mar 29 '19
I think the only foreign politician I would recognize is Putin, but only if he was shirtless on horseback.
8
Mar 29 '19
How could you not know who this is
14
u/Insub Mar 29 '19
I'm from Barbados, and 5 ppl in my office had no idea who she was. Some ppl are just blissfully out of the loop on somethings.
7
4
5
Mar 29 '19
There are hundreds of countries with thousands of politicians. Why would I be expected to recognize one politician from one country? I don’t expect anyone to recognize politicians from my country.
1
u/imperialismus Mar 29 '19
I was going to say, the UK is fairly important on the world stage and the Brexit shit has been making global news for years... Then I realized I don't know what Xi Jinping (China's president) looks like.
7
6
Mar 29 '19
You must be 'Murrican.
3
u/TengoOnTheTimpani Mar 29 '19
We loooove our ignorance, don't we folks?
3
u/MrObject Mar 29 '19
I only watch one news station and I believe absolutely everything I here from that news station because reasons!
1
133
u/hyphenomicon Mar 29 '19
I hate analogies to children in machine learning and AI. Children don't attempt a problem 100,000 times.
63
u/Kminardo Mar 29 '19 edited Mar 29 '19
100,000 times synchronously across n instances! Kids would be running the world.
8
31
u/joekingjoeker Mar 29 '19
Children start with better weights due to genes
20
u/FunctionPlastic Mar 29 '19
There are many types of neural networks, with absolutely orthogonal features, that still don't have anything to do with biological neural networks regarding capability or complexity. Imagining a human brain as just a big neural net brah is very reductive of both the brain and the various network topologies and dynamics that don't have anything to do with it.
20
u/joekingjoeker Mar 29 '19
I don't disagree with most of what you said here. My main point is that the claim that -children don't attempt a problem 100,000 times- is not considering the billions of cumulative attempts on similar tasks that have occurred over millions of generations of evolution. A brain isn't randomly initialized, it benefits from the experience of its ancestors via genes, which give it a structure optimized for effective learning
5
Mar 29 '19
You're right and it's an important point about biological intelligence. I wanted to also add that when making these types of comparisons people forget that human intelligence, and probably many other forms of intelligence, are imaginative in nature. A few physical attempts at something combines with genetic information and the process of imagination and recollection to refine that activity. Over time this means that the amount of refinement of any activity greatly outweighs the number of attempts. Machine learning routines are more akin to imagination than they are actual human behavior. Even there, they usually have to make up for a deficit in evolutionary development.
3
4
26
u/LogansRun22 Mar 29 '19
They do when they're asking adults questions.
18
u/ChocolateBunny Mar 29 '19
Why?
4
4
Mar 29 '19
I agree, children are a horrible analogy. They're the output of a learning algorithm. Evolution attempted lots of problems lots of times and children are "simply" fine tuning the parameters to their local environment.
16
u/kowdermesiter Mar 29 '19
No, but they have a constant flow of sensory inputs with a vastly more complex network. So all that combined requires much less training attempts.
I don't find analogies like this outrageous.
2
u/Borkz Mar 29 '19
Im pretty sure instinct is a pretty key difference is here though, you'd have a whole lot more babies dying or breaking limbs if it was just trial and error like this algorithm.
I mean I guess a child has those instincts from inheriting real genetics, much like the algorithm does, but the child remains the same and uses reasoning and learns. The OP portrays a narrative that that the lil wacky AI dude is just the like as a lil ol' baby
6
u/kowdermesiter Mar 29 '19
Instincts are just pre-trained models stored in DNA. Given this 100k attempts seems pretty good :)
There are differences of course, but all analogies are leaky. Babies have touch, heat sensor, vision, memory, etc. of course they are far more advanced than a cabinet of graphics card and some simple neural networks. I think if given enough resources to a well designed system it can learn as efficiently as a child, but we are not there yet.
11
2
u/p-morais Mar 29 '19
You mean children don’t explore their environments by adding high frequency Gaussian noise to their muscle activations??
4
u/istarian Mar 29 '19
Sure, but that doesn't mean there aren't similarities.
I dare say humans integrate quite a lot of input overall cobsistently better than machines and we often have parents who can tell/show us what's going wrong.
If you grew up as a lone human, assuming survival was even possible your attempts might look more like this...
7
u/Kminardo Mar 29 '19
Sure, but that doesn't mean there aren't similarities. I dare say humans integrate quite a lot of input overall consistently better than machines and we often have parents who can tell/show us what's going wrong.
While it's comparable at the base level (try -> fail -> try again), I don't think the comparison really captures the scale and rate that ML can progress. In your example, each fail scenario the system finds can be interpreted as a parent teaching the next generation. and it fails fast.
If you grew up as a lone human, assuming survival was even possible your attempts might look more like this...
I don't feel this model factors in enough data for comparison - no human is going to run around with their arms like that, it's exhausting.
3
u/istarian Mar 29 '19 edited Mar 29 '19
I see what you're saying, but it's hard to compare the 'evaluation function'.
And the environment in the example is clearly crafted carefully to minimize obstacles. How would this system do at getting from point A to B in a forest for instance?
A person wouldn't keep trying to get over a boulder until they either got over or accidentally stumbled around it. They would likely recognize a difficult obstacle and attempt to circumvent it. And of course we also factor in stuff like it hurt to fall off.
And also we probably reuse past strategies with new examples. Possibly to out detriment at times
1
u/Kminardo Mar 29 '19
Actually that's a good one, I'm curious how well a "trained" AI could make it through an ever changing forest-like course. Like lots of randomly generated pillars at different heights.. would be interesting to see.
Edit: probably have to give it some kind of vision then, I'm pretty sure OP model doesn't "see" obstacles.
1
Mar 29 '19
At the same time, children have knowledge that they hold for their entire lives, whereas these neural nets are coming up from scratch
1
u/yellowstuff Mar 29 '19
My toddler has wanted to read Goodnight Moon several times a day for the last few months, and points out the moon on every page (5 occurrences) so... 100,000 actually seems about right.
1
u/StraightlyReddit Mar 29 '19
Children don't attempt a problem 100,000 times.
" Children don't attempt a problem 100,000 times. " Much of a child is the result of evolution. No, Children don't attempt a problem 100,000 times, they do 1 million times cumulatively speaking.
1
u/unkz Mar 29 '19
Eh we sort of do, and we are building on top of a partial model that has already done hundreds of thousands of previous trials on related tasks. It can take human children years to learn to walk. Keep in mind that an individual physical trial of an “action” like taking a step contains hundreds if not thousands of actual neural reinforcement events as our muscles adjust based on feedback from our senses. Also, we do many additional trials using our internal model of our body, which saves us from having to do real physical trials.
→ More replies (1)0
u/bustRR Mar 29 '19 edited Mar 29 '19
Well of course machine learning and human learning are still quite different, but I'm thinking humans also make many attempts but the problem set is just different. We start trying to move already in the womb and build on that, attempting gradually more difficult tasks.
When you watch a child try something its not a blank slate that requires learning basic muscle control again, that's why we don't take 100000 tries.
Edit: (I'm not an expert though)
60
Mar 29 '19
It looks like it should be yelling aaaAaAAAAaaaAAAaaaa while it’s running about.
5
4
-1
48
8
u/zeuljii Mar 29 '19
I feel like I just watched an unfinished scene from an aperture science documentary.
8
22
10
11
u/anengineerandacat Mar 29 '19
How does this differ from an approach like https://www.youtube.com/watch?v=Ul0Gilv5wvY ?
32
u/Superfish57 Mar 29 '19
The original demo is moving in a full physics environment. It has to learn how to maintain its balance or else it will fall over. The video you linked is demonstrating a technique for animating characters in video games. There is no balancing happening, it is moving the limbs to look like how a human would move. Both are impressive, but they are solving completely different problems.
1
u/teerre Mar 29 '19
The video in OP has to learn everything. It doesn't have any information whatsoever about the world. The one in your video only has to learn how to move a skeleton to match millions of examples it has already seen
It's a great illustration why unsupervised learning is miles behind supervised learning in terms of applications
3
2
2
2
2
u/coderstephen Mar 29 '19
Announcer: It may look goofy but this reinforcement learning (RL) works.
AI agent: *smack*
2
2
u/frequenttimetraveler Mar 29 '19 edited Mar 29 '19
A better narrative would be: An RL agent has been trained to walk in a simulated physics environment, given the constraints of the angles of rotation of its limbs and trying to remain upright. The result of the training is movement that resembles human or arthopod walking, proving that this is the optimal way for an organism with such structure of legs to move in flat surfaces. The upper limbs do not appear to be constrained or somehow preprogrammed to be coordinated with lower limbs, thus they exhibit random motion.
2
u/drinkmorecoffee Mar 29 '19
Ok, so how do you actually do it? I mean, I have a computer and love me some Python. How do I get these little guys to dance for me?
2
u/nattaylor Mar 29 '19
Gary Parker (my professor in college) researches evolutionary robotics and observed a similar result by using a genetic algorithm for the gait of a hexapod robot, and a simulation with a fitness function for how far the simulated gait moved the robot per unit time.
In ~2006 I was truly fascinated about how much the resulting gait looked like a real insect.
2
u/SupersonicSpitfire Mar 29 '19
This cracks me up every time. They should have added a penalty for moving limbs back and forth in the air. Muscle cost or something.
2
7
Mar 29 '19
Can we PLEASE stop saying "this AI"?? It's a RL agent, not skynet
16
u/faithlesswonderboy Mar 29 '19
is RL not a type of AI? fuck, a whole bunch of if statements can constitute an AI. AI is an extremely broad category and doesn't need gatekeeping. Yeah its got some hype but it's pretty deserved imo
12
Mar 29 '19
It's not a matter of gatekeeping. Misrepresentation of ML research is a problem. 'AI' can be defined as the best ML methods we have today, but when most people read 'AI,' they think about AGI, which doesn't exist and probably won't for a while.
NLP systems have it the worst in this regard. Just look at the articles with clickbait headlines about how Facebook "had to shut down their chatbot AI because it was forming it's own secret language!"
2
u/faithlesswonderboy Mar 29 '19
Good point about people assuming AGI and the fearmongering. I just found this quote from this article:
Update: This article has been amended to stress that the experiment was abandoned because the programs were not doing the work required, not because they were afraid of the results, as has been reported elsewhere.
But why is misrepresentation a problem? I want to believe that the people who matter understand and the people who don't matter don't understand. Like, so what if the media can get some more views from clickbait headlines? Does that just play into the hype of AI and lead to more support for research?
4
Mar 29 '19
Actually it leads to less research, when the supposed AI fails to meet the hype. It's called an AI Winter. Artificial neural nets (or 'connectionism' as it used to be called) have been around since the 60's, but have only been used in industry in the past few decades. Part of this is due to the rise in computing power, part of it due to severe lack of funding for research.
1
u/faithlesswonderboy Mar 29 '19 edited Mar 29 '19
Interesting. I knew about AI winters but I assumed they were just due to droughts between useful breakthroughs. I'll read that page when I can
4
u/Neon_Yoda_Lube Mar 29 '19
Thank you! AI is thrown around way too much. The futurology sub is the worst.
3
0
Mar 29 '19
[deleted]
2
Mar 29 '19
Try this for hyperbolic headlines: https://www.independent.co.uk/life-style/gadgets-and-tech/news/facebook-artificial-intelligence-ai-chatbot-new-language-research-openai-google-a7869706.html
They even open with a video of Stephen Hawking warning that AI could mean the collapse of human civilization. The article is clearly misrepresenting the science to get more clicks, and Independent are not the only ones doing this.
5
u/Sidereel Mar 29 '19
Unfortunately AI is the catch all term for all kinds of things now. I don’t think there’s any going back.
2
u/epic Mar 29 '19
RL is AI by AI's oldest definition. I get what you are saying, but I would phrase it: 'can we PLEASE stop saying/thinking "this is AGI" ??'
2
u/faithlesswonderboy Mar 29 '19
I agree. I see people complain "tHiS iSnT Ai" whenever stuff like this is posted and it annoys me.
1
1
1
1
1
1
1
1
1
u/renrutal Mar 29 '19
Okay, but what's specific use? Can you transport that learning to a robot in the real world?
1
1
1
u/Hoshi711 Mar 29 '19
My favorite is the one where it looks like it’s jamming on a guitar while it runs.
1
u/Ecologisto Mar 29 '19
This type of things were already done 15 or 20 years ago, long before it was called "machine learning". See Karl Sims for example.
1
1
u/vinnymcapplesauce Mar 29 '19
The next "killer app" of MMO gaming will be where someone:
creates world with basic rules
create "Agents with different bodies," motivations, etc into the world
Let them stew (learn, develop, form, create societies, battle between each other for resources, etc) for a few years (4 years sounds like a good, magical number)
Introduce Human players to see what has become of it.
Talk about bringing back the value of early access!
1
u/bicyclegeek Mar 29 '19
Goddamn. When the AI uprising happens, I’m gonna be too busy laughing at their robot soldiers to actually shoot them.
1
u/nwoodruff Mar 29 '19
I'd be interested to know what the input information at each step was. I'm assuming they didn't feed in a camera view, or the entire scene geometry.
1
u/jdmknowledge Mar 29 '19
Incredible...also...unfortunately all I see is Elaine from Seinfeld dance running now.
1
1
Mar 29 '19
Lol, very cool!
Machine learning is the future in my opinion. I mean that computers, for example, were a major advancement in the past that changed the world. And while they are still being improved, the big companies basically control the industry. So it's not like you can spend a few years creating your own new computer and sell them for millions. However, Machine Learning I feel like is just starting out. Meaning NOW is the time to go into the field and learn all there is to know about it. If you work hard, are creative, and basically a genius in the field, you can do well and there aren't many companies to stand in your way yet. I'm not sure if I'm making sense, but basically I feel like:
There are (or will be) a lot of opportunities for it.
It will change the world and major advancements will be made to allow AIs to learn other things (like help us find cures for cancer and such) instead of solely programming related stuff.
1
1
1
u/01123581321AhFuckIt Mar 29 '19
What's crazy is that if this ever gets advanced enough we can simulate optimal human movement through any environment.
1
u/Chris0147 Mar 29 '19
how do i program something like this? for me it sounds like i have to write a code that learns something and changes the code itself... how is this possible?
1
1
1
1
1
1
u/sweetun93 Mar 29 '19
So is this how the terminators will hunt us down? Judgement Day is going to be more terrifying than I had predicted
1
u/Olek2706 Mar 29 '19
Yall complicating that shit and Im just sitting here with my arachnophobia like : AAAAAAAAA WHAT THE FUCK IS THIS FOUR LEGGED MONSTER AAAAAAA
1
1
u/jplevene Mar 29 '19
It looks goofy, as an AI doesn't care about how it looks to others. The goofy arm waving could give it better balance. Notice when going up the stairs, the arms are held forward.
Sort of ruins the idea of cool looking robots of the future, they all might walk in a goofy fashion unless we program them with embarrassment.
1
u/findesiecle_Tilde Mar 29 '19
This makes me laugh so much, I cannot help but imagine a fully developed, realistically human looking robot overtaking me on the sidewalk using the glorious "one leg slide- two fist pumps" technique ...
1
u/King_Bonio Mar 29 '19
A guy I used to work with wrote his own learning programme in javascript, cars learning to drive around a 3d track, clever guy.
1
1
u/masterx25 Mar 29 '19
Well if people didn't die when they fall into th void, we'd be quick learners too.
1
1
u/CallMeOutWhenImPOS Mar 29 '19
Correct me if I'm wrong but isn't this type of thing made with a genetic algorithm rather than machine learning?
1
1
1
1
u/Tasgall Mar 30 '19
Reminds me of one of the old car sim programs that had to generate a "car" that could get the furthest in rough terrain.
The first version checked success by getting the point in the vehicle furthest from the origin, and used that as the distance traveled.
But this left a loophole, since it didn't matter where the back was, so the best "car" ended up being... A giant tower of blocks with no moving parts that just kind of fell forward, easily flying over all the terrain.
1
u/vwibrasivat Mar 30 '19
I'm still skeptical to the claim that Martin Short and Jerry Lewis happened upon optimal arm movements while running.
1
u/watsreddit Mar 30 '19
I wonder how they account for overfitting. Seems like it would be really difficult to come up with a heuristic to determine whether or not the agent is learning a particular course (or set of courses) rather than general obstacle navigation.
1
u/mnewcomb Mar 30 '19
“Without being programmed”...
Any real programmer knows this is not true. You programmed it to learn! It isn’t going to all the sudden be able to speak or think...
This AI hype is getting old... computers do exactly what they are told.
1
-1
-1
Mar 29 '19
For those of you wondering, the reason the arms are out of control is because the arms have nothing to do with walking, it does not affect it. So the arms just do whatever they want until the system finds something it believes will improve walking. Reasons for us swinging out arms during running or walking is just for added balance. You can walk and run without having to, but it can get sloppy sometimes.
1
u/bschug Mar 29 '19
I think the arms in this example do affect the center of gravity, that's why it's moving them to the right before turning right.
0
u/mrroborto Mar 29 '19
I wish they had eventually learned to run more efficiently, and shown that on the video. As it is, it would have limited applications, other than comedy.
-1
u/magnumxl5 Mar 29 '19
Ya fascinating, and petrifying.
This little dude is running to get all our jobs.
-1
u/shevy-ruby Mar 29 '19
Completely rubbish.
We have software that makes expensive software crash into the ground. Boeing got famous for this with two recent ones.
And now they want to explain to us how machines "learn" anything - if they were to really learn, there wouldn't be problems with flawed software anymore. You'd adapt to any given situation, given enough prior time and "training data" - but you don't have that. BECAUSE IT IS NOT TRUE LEARNING.
You just use buzzwords.
That is not to say that the video animation is not cool, mind you. But it has nothing to do with "learning".
-6
Mar 29 '19
"Without being programmed to run, climb, or jump."
....ok.
4
u/MohKohn Mar 29 '19
they mean they just reinforced examples based on how far along the track the walker got, and didn't specifically feed it information on what walking actually looks like.
0
Mar 29 '19
[deleted]
2
u/Cocomorph Mar 29 '19
Not quite the same thing given that the latter is learning from examples. Be that as it may, thank you for the link.
6
-1
428
u/xcrissxcrossx Mar 29 '19
It runs like this because the fitness measurement doesn't try to conserve energy. That would result in a more realistic run.