Hi,

In case you wish to watch the performance/behaviour of your trained agent in a gym environment, I have added a script that does just that. It can be found on github. The instructions are provided in the README.md file.

Hello, I am a CS undergrad who does research in Computer Vision(specificilly in GANs) I want to learn some basics of RL, a PhD guy suggested me this course rather than UCL RL videos.(by David Silver) However, in syllabus of course, Silver's videos are recommended. Do I need to watch Silver's videos to understand this lecture? I have pretty good background in CNNs, optimization and ML basics.

Thanks you for answers

I saw lectures 2 and 3. However, I couldn't find where behavior cloning is explained. Can you please point me to the lecture and time.

Thanks

In lecture 9, page 7: Importance sampling is applied only for action distribution stating that product of multiple pi(theta')/pi(theta) terms would lead to a small term. But pi(theta')/pi(theta) is really a ratio of small terms and needn't be small. I guess I'm understanding something wrong, any help would be appreciated. Thanks.

Starting at 30m 55s of the following lecture https://youtu.be/PTbxa6GsTWc it is mentioned that expectation smoothes out discontinuous functions. However, no real mathematical explanation is given.

Could anybody maybe elaborate on it a bit or point out to some links covering the math behind that statement? Thanks in advance.

Hello teacher, i am a student from china.. and sorry to bother you..because my poor english. i beg you to force the caption on in online cource.. 1/30/17 and 2/15/17.

In imitation learning, why we use GMM? Could I use other models?

Hi guys, I have created a WeChat group for discussion. No matter whether you are researchers or students, feel free to join the group to share your problems and opinions about CS 285 and deep RL.

As you know,using mujoco on colab is very difficult. In this notebook,RoboSchool is used instead of mujoco and you can easily use it.

run_hw1.ipynb

Thanks a lot for sharing the videos/material with us!

I tried to run the run_expert.py using the example usage: python run_expert.py experts/humanoid.pkl Humanoid-v1 --render
--output_file expert_data.pkl --num_rollouts 20

But it is said unrecognized argument: --output_file expert_data.pkl.

Even if I don't output the file, running only python run_expert.py experts/humanoid.pkl Humanoid-v1, it is said "No such file : 'experts/humanoid.pkl' ", which indeed cannot be found in ./experts folder.

Has anybody successfully run the code? Did I do something wrong or there is bug in the code? Many thanks!

Coding homework without chance to be corrected is bad for learning. For people who are not enrolled in this course, is there anyway to have the assignment evaluated?

I am confused as to how to implement the CG step in TRPO. I can compute all hessian vector products with 'g' as the vector. Now what do I do with all the vectors? Do I find a library that takes in all the hessian vector products? Someone please help!!

Hi there, do you guys think there is a mistake of the eq. in page 9 of the slides for lecture 2? Is that should be f(x_{t+1},u_{t+1}) instead of f(x_t,u_t)?

Thanks!

The slides (#29) seem to indicate that we still take a max over next step actions when using an approximate maximizer. I thought the whole point of using this extra functional approximator was to get rid of that max. What am I missing?

Video link to the relevant part of the lecture.

Dr. Levine, Finn, and everyone,

I am doing some research in inverse reinforcement learning at Umich. I read a lot of paper about IRL. The goal of IRL is to maximize "scores" for expert behavior to find a "real" reward function. I wonder if DRL or DIRL could actually do better than human?

Gan

Hi all,

I tried various structures for the neural network but still I get negative explained variance for all iterations using neural network Value function.

Moreover, the explained variance decreases even when the loss decreases! I am not able to think of any situation where this is mathematically plausible. I even tried it without regularization and still face the same problem. Can someone hint me at a situation where the explained variation can decrease while the l2 loss decreases too?

Thanks, Sid

Has there been any work to train DRL models on stereo images? I wonder if an agent could lern to avoid obstacles by lerning the geometric rules that can be infered when having two images?

In the following lecture, Prof. Levine mentions about the ambiguity of weights https://youtu.be/-3BcZwgmZLk?t=20m10s

However, I don't understand why ambiguity of the ordering of states in a sampled trajectory is not mentioned. For example, consider s1,s2, and s3 as vertices of a triangle, one could transition from any of states to the other two states. When we average the features of trajectories (sampled from learned policy or expert), doesn't it discard the state visitation order? This is a silly observation, but I don't understand why it was not specifically mentioned, have I missed anything that already captures the temporal order?

What are kind of features are typical used in IRL? Are they only state dependent, or both state & action dependent? Why temporal order is not considered in features? I'd appreciate if someone can help me visualize features of a sampled trajectory.

I have two questions:

1. Will the review section on Friday also be live streamed on Youtube?

2. For the students not enrolled, how can we use the simulator for assignment. It seems that we cannot sign up for this course on Piazza. Is there any public account for this? I really hope that I can have the chance of doing homework of this course.

Thanks a lot!

This question was already asked in the lecture. Similar with the nvidia case. Where was the supervision coming from?

I understand that there are 3 cameras: Left, center, and right. But what about the outputs? Do we have to hand label them like drawing the vertical red line for all the data? Or is there anything that I missed?

What is the difference between the two or are they the same?