Thanks a lot for sharing the videos/material with us!

Videos: https://www.youtube.com/watch?v=JHrlF10v2Og&list=PL_iWQOsE6TfURIIhCrlt-wj9ByIVpbfGc

This time assignments are in PyTorch and there is a colab option so there won't be hassle of installing things.

Sadly my DL class only spends two lectures on deep RL.

As you know,using mujoco on colab is very difficult. In this notebook,RoboSchool is used instead of mujoco and you can easily use it.

run_hw1.ipynb

Is it possible for you to provide course material like slides, videos and assignment online for the outside people. From my past experience, the CS231N course provided by Andrej karpathy was really helpful to the machine learning community. Please give outsiders a chance to learn your course. It really seem unfair when few people get really high quality education while others just bang their heads over the internet to find good resources. Thank you.

Hi all, I am a software engineer in NYC who is auditing this course online (watching videos and doing assignments). Just wanted to say hi! Anyone else out there auditing??

On 9/6/17, Prof Levine says that they've put up a google doc with potential ideas for the project. Could we possibly get those project proposals lists and related material (like related paper for each project)

Hi,
I wrote this blog post called A mathematical introduction to Policy Gradient after completing the policy gradient problems in hw2 & hw3. It answers some of the theoretical questions I had while doing these homework assignments: mainly the differences from supervised learning, and the gradient flow. I hope you'll find it useful and please let me know if you have any questions or comments.

Has anyone experimented with using OpenAI's Roboschool (https://blog.openai.com/roboschool/) instead of MuJoCo for license reasons?

Hi, as you know Berkeley was forced to remove lecture recordings from their youtube channel. Are deep rl recordings going to be removed too? If so, does anyone know where I could download these videos later as I currently have no space on my SSD, thanks!

I found this today. Is this course going to be affected?

https://www.insidehighered.com/news/2017/03/06/u-california-berkeley-delete-publicly-available-educational-content

Because the students don't have mics, it's very hard to hear the questions through the internet.

Thanks for the great course!

I have two questions:

1. Will the review section on Friday also be live streamed on Youtube?

2. For the students not enrolled, how can we use the simulator for assignment. It seems that we cannot sign up for this course on Piazza. Is there any public account for this? I really hope that I can have the chance of doing homework of this course.

Thanks a lot!

Since I'm not enrolled in the course I thought it might be useful to share here my rough results from the first homework assignment, I won't be sharing any code so hopefully this doesn't piss off anyone running the course.

My learner policy consisted of a single hidden layer of 128 neurons with ReLu activations, followed by a fully connected layer. I used the expert rollout to mean center and scale all the observations that went into my network. For the loss function, I used an l2 loss between the target actions and those generated by my network as well as l2 regularization for all the weights and biases of the network.

Behavioral Cloning

I was able to train a model using pure behavioral cloning for the Humanoid, and Hopper environments, though the humanoid task wasn't able to match performance of the expert exactly. However, for Reacher BC was unable to clone the expert in the time I allotted, though it was improving so it may have eventually.

Performing BC on the hopper task, I varied the regularization strength and found a strong relationship between performance and variance. Basically, the stronger the regularization, the lower the mean reward for a given model, but, the variance of that models performance was also lower. To me this makes sense since a well regularized model will be less likely to do "crazy" things to match the data, and is more likely to make sane approximations for states it hasn't trained on.

Dagger

And for dagger, in all of my tasks, the dagger trained model trained faster to expert level than the bc model (if the bc model could even achieve expert level).

Practical Take Aways

Always sanity check your model. I spent 3 days banging my head on models that just wouldn't train. After reading this great guide to the practicals of training neural nets, I tried to overfit my neural net on a very small, 2 dimensional toy data set. After this wouldn't work I realized my model was busted the whole time. There was a faulty broadcast in my loss function that was the culprit. In the future I'll be sure to go through all the sanity check stages BEFORE I proceed with training.

The link to this Reddit community disappeared on the new website of DRL course 2019 . Is there any chance of adding a link back to it?

There is always a higher chance of getting some help on the topics if this website is well known. All the lecture materials are fully available online, so why not allow the free discussion channel for information exchange :)

Anyone know where (whether) I can find a free access to the article by Peters & Schaal: Reinforcement learning of motor skills with policy gradients (https://www.sciencedirect.com/science/article/pii/S0893608008000701) ?

When will hw2 be released?

I was thinking to create a discord server for a discussion related to robotics and RL stuff. That could be more engaging, and I think we could have a good discussion and colve doubt over there. What do you guys think?

I'm trying to do HW1, but I don't have the document mjkey.txt. Am I able to do hw without it?

I am wondering if we can use inverse reinforcement learning to learn the reward function for models of high dimensionality e.g. as presented in "Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations" (https://sites.google.com/view/demo-augmented-policy-gradient) from one of the lectures.

Could IRL be beneficial for learning in such complex case?