DDQN

This post documents my implementation of the Double Deep Q Network (DDQN) algorithm.

A Double Deep Q Network (DDQN) implementation in tensorflow with random experience replay. The code is tested with Gym’s discrete action space environment, CartPole-v0 on Colab.

Code on my Github

If Github is not loading the Jupyter notebook, a known Github issue, click here to view the notebook on Jupyter’s nbviewer.

Notations:

Model network = \(Q_{\theta}\)

Model parameter = \(\theta\)

Model network Q value = \(Q_{\theta}\) (s, a)

Target network = \(Q_{\phi}\)

Target parameter = \(\phi\)

Target network Q value = \(Q_{\phi}\) (\(s^{'}\), \(a^{'}\))

Equations:

TD target = r (s, a) \(+\) \(\gamma\) \(Q_{\phi}\) (\(s^{'}\), \(argmax_{a^{'}}\) \(Q_{\theta}\) (s\(^{'}\), a\(^{'}\)))

TD error = (TD target) \(-\) (Model network Q value) = [r (s, a) \(+\) \(\gamma\) \(Q_{\phi}\) (\(s^{'}\), \(argmax_{a^{'}}\) \(Q_{\theta}\) (s\(^{'}\), a\(^{'}\)))] \(-\) \(Q_{\theta}\) (s, a)

Key implementation details:

Create a placeholder to feed Q values from model network:

self.model_s_next_Q_val = tf.placeholder(tf.float32, [None,self.num_actions], name='model_s_next_Q_val')

Select Q values from model network using \(s^{'}\) as features & feed them to the training session:

# select actions from model network
model_s_next_Q_val = self.sess.run(self.model_Q_val, feed_dict={self.s: s_next})

# training
_, loss = self.sess.run([self.optimizer, self.loss],
                        feed_dict = {self.s: s,
                                     self.a: a,
                                     self.r: r,
                                     self.s_next: s_next,
                                     self.done: done,
                                     self.model_s_next_Q_val: model_s_next_Q_val})

Select minibatch actions with largest Q values from model network, create indices & select corresponding minibatch actions from target network:

def td_target(self, s_next, r, done, model_s_next_Q_val, target_Q_val):
    # select action with largest Q value from model network
    model_max_a = tf.argmax(model_s_next_Q_val, axis=1, output_type=tf.dtypes.int32)

    arr = tf.range(tf.shape(model_max_a)[0], dtype=tf.int32) # create row indices
    indices = tf.stack([arr, model_max_a], axis=1) # create 2D indices        
    max_target_Q_val = tf.gather_nd(target_Q_val, indices) # select minibatch actions from target network
    max_target_Q_val = tf.reshape(max_target_Q_val, (self.minibatch_size,1))

Tensorflow graph:

References:

Deep Reinforcement Learning with Double Q-learning (Hasselt, Guez & Silver, 2016)

• 2020 10
• 2019 24