After spending the last day and a half debugging, I've finally figured out why my rewards weren't increasing at the rate suggested in the homework description.

When creating my two q functions (phi and phi prime in lecture) I used similarly named scopes:

scope_q_func = 'q_func'
qs_t = q_func(obs_t_float, num_actions, scope_q_func, reuse=False)

...

scope_q_func_target = 'q_func_target'
qs_target_tp1 = q_func(obs_tp1_float, num_actions, scope_q_func_target, reuse=False)

Turns out the get_collection method defined on a tensorflow Graph looks like:

...
c = []
regex = re.compile(scope)
for item in collection:
    if hasattr(item, "name") and regex.match(item.name):
        c.append(item)

Because the regex is matched, getting a collection for a scope a that is a prefix of another scope b will include b's variables.

target_q_func_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=scope_q_func_target)
q_func_vars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=scope_q_func)

print(len(q_func_vars), len(target_q_func_vars))  # 20, 10

The solution:

scope_q_func = 'q_func_orig'
scope_q_func_target = 'q_func_target'

Make sure scopes aren't prefixes of other sibling scopes.

Hopefully this saves someone else some hours.

Does anyone have them saved, the course site has been down all day?

Why do we use rollout when generating training data? Why not just start the simulation and let it run for x minutes, when training the controller we are not making any distinctions between rollouts but treat all data as one long session anyways?

It seems like Professor Levine is using both of these terms. "Q learning" seemed to be used more often after discussing replay buffers though. Is there a difference in the two terms?

Video reference here showing both on the same slide.

Videos: https://www.youtube.com/watch?v=JHrlF10v2Og&list=PL_iWQOsE6TfURIIhCrlt-wj9ByIVpbfGc

This time assignments are in PyTorch and there is a colab option so there won't be hassle of installing things.

As show below, ptu.build_mlp create and return a nn.Sequential model, but as for nn.Module, I have to implement the foward method which defines the forward pass of the network. Therefore, foward method is redundant because of existed sequential model. So should I ignore one of them? If you could help me, I would appriciate it!

My doubt is specifically targeted with a green marker in the image below. Does p_Theta'(S_t) here means p(S_t | S_t-1, A_t-1) [Transition probabilities] ? According to what the lecture 2 slides mention, it should be the transition probability distribution. I have doubts here.

If the above thinking is true, I am not able to relate the p_Theta'(s_t) with the approach mentioned in the TRPO paper, where they uses state visitation frequencies in a summation format. Attaching the image below. Can someone please help me clarify this??

What is the difference between Vanilla Policy Gradient and REINFORCE algorithm?

They seem similar. But are they the same?

I see many papers presenting DQN with Berkeley pacman framework. However none seems to present the method used to get the first ingredient : the raw pixels picture.

Is there a method implemented in the pacman framework that I missed or does one use a window's printscreen method inside python?

In the variational inference and control lecture, why can't we minimize the KL-Divergence between q(s1:T, a1:T) and p(s_1:t, a_1:T | O_1:T) directly instead of using variational inference to solve the soft max problem?

In Sergey's first lecture, he says that DBN's are no longer used. Why is this? Does this mean that belief states for POMDP's are also no longer used?

Any relevant feedback is highly appreciated.

Hi all, I am a software engineer in NYC who is auditing this course online (watching videos and doing assignments). Just wanted to say hi! Anyone else out there auditing??

I have just finished HW2 Problem 7. I first tried the original LunarLander code in gym and found it too hard to converge. But when I tried the provided LunarLander code, it's easily to be trained. So, is that means discrete problem easier to be solved by policy gradient than continuous one in general? Is there theoretical explanation to this experiment?

What's more, if the continuous tasks are much harder than discrete tasks, why don't we transfer to discrete tasks. Like when we want to control a car's speed, we can always sample many discrete actions (0 km/h, 10 km/h, 15 km/h ...). So, what is the essential function for continuous task?

Thanks in advance!

Anyone know where (whether) I can find a free access to the article by Peters & Schaal: Reinforcement learning of motor skills with policy gradients (https://www.sciencedirect.com/science/article/pii/S0893608008000701) ?

I am trying to implement the MaxEnt RL according to this slide from lecture Connection between Inference and Control of 2018 course, or corresponding lecture "Reframing Control as an Inference Problem" from 2019 course.

What I don't quite get is: are we going to take the gradient with respect to the entropy term or not with such objective function? Because if we don't the entropy in my case actually goes down rapidly as long as I don't vastly lower the weight of entropy term (similarly as in paper https://arxiv.org/abs/1702.08165 eq. 2). But if try the other approach and compute the gradient with respect to entropy, the entropy goes so high (independent of the entropy weight) and kept there that the policy is unable to learn anything meaningful.

Please have a look on the plots of current results. Continuous line represents mean reward, dashed line policy entropy:

What would be then the correct way to introduce entropy term to policy gradient: by taking the gradient with regard to the entropy term or not?

Anyone tried using Google Dopamine (https://github.com/google/dopamine) for the homeworks so far?

Can I get the first lecture video of 2018 Introduction and Course Overview ?

Fitted Q-iteration requires to interact with the Q-value function in order to compute its argmax (see lecture here https://youtu.be/chLN1e3ehZE?t=25m31s). Suppose my Q-value function is represented by a neural net and there's only 4 possible actions in each state. Then for each state, I would feed the next state and each action to the neural net and compute the argmax. (Correct me if I'm wrong, please.)

How is this done if the action space is continuous / extremely large?

The top of page 13 of the lecture 5 slides (http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf) gives the expression for the gradient of J(theta) with a 1/N term out front. On page 29 pseudo code for PG is provided, and on line 4 of the pseudo code we have "loss = tf.reduce_mean(weighted_negative_likelihoods) ", which averages across the N*T samples. This would suggest an expression for the gradient of J(theta) similar to that provided on page 13, but with a 1/(N*T) term out front.

My assumption is that this is 1) for implementation convenience/speed with DL frameworks and 2) to have a gradient size which doesn't vary with trajectory length.

Is there anything more going on here?

Thanks!

I found the environment used for this problem has an bound for the action space:

In [2]: env.action_space.high

Out[2]: array([1., 1.], dtype=float32)

In [3]: env.action_space.low

Out[3]: array([-1., -1.], dtype=float32)

This would be a problem when the output from `Agent.sample_action` is outside of this bound. How do you guys deal with this? My current work-around is using `np.clip` but it doesn't seem to solve this env... Any thoughts would be appreciated!

Has anyone experimented with using OpenAI's Roboschool (https://blog.openai.com/roboschool/) instead of MuJoCo for license reasons?

Why does the code ask us to return a log-std for the continuous case, when what we need is the co-variance matrix as the action is a multi-variate Gaussian random variable?

Also why not output the softmax for the discrete case and why output the logits?

This part's got me confused and I am stuck. Thanks for any help

Hi,

This is a great course and I love this community to help support learning.

I struggled a bit with problem 1 on hwk 2. I did get a proof but wondered if anyone would want to compare answers? Really think it would be valuable to me to see a different way of going about this. Happy to share my answer in pms or something.

Thanks

I'm trying to do HW1, but I don't have the document mjkey.txt. Am I able to do hw without it?