DQN

This post documents my implementation of the Deep Q Network (DQN) algorithm.

A Deep Q Network implementation in tensorflow with target network & random experience replay. The code is tested with Gym’s discrete action space environment, CartPole-v0 on Colab.

Code on my Github

If Github is not loading the Jupyter notebook, a known Github issue, click here to view the notebook on Jupyter’s nbviewer.

Notations:

Model network = \(Q_{\theta}\)

Model parameter = \(\theta\)

Model network Q value = \(Q_{\theta}\) (s, a)

Target network = \(Q_{\phi}\)

Target parameter = \(\phi\)

Target network Q value = \(Q_{\phi}\) (\(s^{'}\), \(a^{'}\))

Equations:

TD target = r (s, a) \(+\) \(\gamma\) \(max_{a}\) \(Q_{\phi}\) (\(s^{'}\), \(a^{'}\))

TD error = (TD target) \(-\) (Model network Q value) = [r (s, a) \(+\) \(\gamma\) \(max_{a^{'}}\) \(Q_{\phi}\) (\(s^{'}\), \(a^{'}\))] \(-\) \(Q_{\theta}\) (s, a)

Key implementation details:

Update target parameter \(\phi\) with model parameter \(\theta\). Copy \(\theta\) to \(\phi\) with either soft or hard parameter update.

Hard parameter update:

with tf.variable_scope('hard_replace'):
  self.target_replace_hard = [t.assign(m) for t, m in zip(self.target_net_params, self.model_net_params)]   

# hard params replacement
if self.learn_step % self.tau_step == 0:
    self.sess.run(self.target_replace_hard)  
self.learn_step += 1

Soft parameter update: polyak \(\cdot\) \(\theta\) + (1 \(-\) polyak) \(\cdot\) \(\phi\)

with tf.variable_scope('soft_replace'):            
  self.target_replace_soft = [t.assign(self.polyak * m + (1 - self.polyak) * t)
                              for t, m in zip(self.target_net_params, self.model_net_params)]   

Stop TD target from contributing to gradient computation:

# exclude td_target in gradient computation
td_target = tf.stop_gradient(td_target)

Tensorflow graph:

References:

Human-level control through deep reinforcement learning (Mnih et al., 2015)

• 2020 10
• 2019 24