Measurement & mixed states for quantum systems.
Notes on measurement for quantum systems.
RNN backprop thru time(BPTT)
Notes on the math for RNN back propagation through time(BPTT).
\(\hat{y}\) is the prediction.
\(U\) is the weight matrix after the hidden layer.
\[\hat{y} = f_y (U h_t + b_y)\]\(h_{t}\) is the hidden layer at time t.
\(f_{h}\) is the non linear function in the hidden layer.
\(V\) is the weight matrix before the hidden layer.
\(W_{h_{t-1}}\) is the weight matrix in the hidden layer at the previous time step.
\(b_{h}\) is the bias at the hidden layer.
\[h_{t} = f_{h} (Vx_t + Wh_{t-1} + b_{h})\]No need to BPTT for this:
\[\frac{\delta L_{t}} {\delta U} = \sum_{i=0}^{T} \frac{\delta L_i} {\delta U} = \frac{\delta L_t} {\delta \hat{y}_t} \frac{\delta \hat{y}_t} {\delta U}\]BPTT
The loss at time t with respect to the weights in the hidden layer.
The terms in square brackets \([]\) are written in such a way that the leftmost term is the most recent term while the rightmost is the oldest term.
When k = t, the product sequence (or factor) on the left of \(\frac{\delta h_{t}} {\delta W}\), equals 1.
\[\frac{\delta L_{t}} {\delta W} = \frac{\delta L_{t}} {\delta \hat{y}_{t}} \frac{\delta \hat{y}_{t}} {\delta L_{t}} [ (1) \frac{\delta h_{t}} {\delta W} + ( \frac{\delta h_{t}} {\delta h_{t-1}} \frac{\delta h_{t-1}} {\delta W} ) + ( \frac{\delta h_{t}} {\delta h_{t-1}} \frac{\delta h_{t-1}} {\delta h_{t-2}} \frac{\delta h_{t-2}} {\delta W} ) + ... ]\] \[= \frac{\delta L_t} {\delta \hat{y}_t} \frac{\delta \hat{y}_t} {\delta h_t} \sum_{k=0}^{t} [ \frac{\delta h_{t}} {\delta h_{t-1}} \frac{\delta h_{t-1}} {\delta h_{t-2}} ... \frac{\delta h_{k+2}} {\delta h_{k+1}} \frac{\delta h_{k+1}} {\delta h_k} \frac{\delta h_k} {\delta W} ]\] \[= \frac{\delta L_t} {\delta \hat{y}_t} \frac{\delta \hat{y}_t} {\delta h_t} [ \sum_{k=0}^{t} ( \prod_{i=k+1}^{t} \frac{\delta h_{i}} {\delta h_{i-1}} ) \frac{\delta h_k} {\delta W} ]\]\[\frac{\delta L_{t}} {\delta V}\]
BPTT also needs to be done for calculating the loss with respect to weight matrix V. The dependence between hidden units and weight matrix V is not only in one place. Hidden units from all the previous time steps also depend on V so we need to go backwards in time to calculate this gradient.
2020
Quantum States as a generalization of classical probabilities.
Notes on quantum states as a generalization of classical probabilities.
Finding the ray_results folder in colab
The location of ray_results folder in colab when using RLlib &/or tune.
PBT for MARL
My attempt to implement a water down version of PBT (Population based training) for MARL (Multi-agent reinforcement learning).
RLlib trainer common config
Ray (0.8.2) RLlib trainer common config from:
Output dimension from convolution layer
How to calculate dimension of output from a convolution layer?
Changing G drive directory in Colab
Changing Google drive directory in Colab.
Linear regression (Bayesian)
Notes on the probability for linear regression (Bayesian)
RNN backprop thru time(BPTT part 2) \(\frac{\delta h_{t}} {\delta h_{t-1}}\)
Notes on the math for RNN back propagation through time(BPTT), part 2. The 1st derivative of \(h_t\) with respect to \(h_{t-1}\).
RNN backprop thru time(BPTT)
Notes on the math for RNN back propagation through time(BPTT).
2019
Filter rows with same column values in Pandas dataframe
Filter rows with same column values in a Pandas dataframe.
Custom Sagemaker reinforcement learning container
Building & testing custom Sagemaker RL container.
Reinforcement learning custom environment in Sagemaker with Ray (RLlib)
Demo setup for simple (reinforcement learning) custom environment in Sagemaker. This example uses Proximal Policy Optimization with Ray (RLlib).
Django + Postgres + Travis CI + Heroku CD
Basic workflow of testing a Django & Postgres web app with Travis (continuous integration) & deployment to Heroku (continuous deployment).
Django + Postgres + Docker + Travis CI + Heroku CD
Basic workflow of testing a dockerized Django & Postgres web app with Travis (continuous integration) & deployment to Heroku (continuous deployment).
Dockerized Postgres connection with Django web app in Travis CI
Introducing a delay to allow proper connection between dockerized Postgres & Django web app in Travis CI.
Random policy in RLlib
Creating & seeding a random policy class in RLlib.
Custom MARL (multi-agent reinforcement learning) CDA (continuous double auction) environment
A custom MARL (multi-agent reinforcement learning) environment where multiple agents trade against one another in a CDA (continuous double auction).
Tensorflow graphs in Tensorboard
This post demonstrate how setup & access Tensorflow graphs.
.bash_profile for Mac
This post demonstrates how to create customized functions to bundle commands in a .bash_profile file on Mac.
RND (Random Network Distillation) with Proximal Policy Optimization (PPO) Tensorflow
This post documents my implementation of the Random Network Distillation (RND) with Proximal Policy Optimization (PPO) algorithm. (continuous version)
DPPO distributed tensorflow
This post documents my implementation of the Distributed Proximal Policy Optimization (Distributed PPO or DPPO) algorithm. (Distributed continuous version)
A3C distributed tensorflow
This post documents my implementation of the A3C (Asynchronous Advantage Actor Critic) algorithm (Distributed discrete version).
A3C multi-threaded continuous version with N step targets
This post documents my implementation of the A3C (Asynchronous Advantage Actor Critic) algorithm. (multi-threaded continuous version)
A3C multi-threaded discrete version with N step targets
This post documents my implementation of the A3C (Asynchronous Advantage Actor Critic) algorithm (discrete). (multi-threaded discrete version)
Accumulate gradients with Tensorflow
This post demonstrates how to accumulate gradients with Tensorflow.
Distributed Tensorflow
This post demonstrates a simple usage example of distributed Tensorflow with Python multiprocessing package.
N-step targets
This post documents my implementation of the N-step Q-values estimation algorithm.
Python’s multiprocessing package
This post demonstrates how to use the Python’s multiprocessing package to achieve parallel data generation.
Numpy array manipulation
This post provides a simple usage examples for common Numpy array manipulation.
Dueling DDQN with PER
This post documents my implementation of the Dueling Double Deep Q Network with Priority Experience Replay (Duel DDQN with PER) algorithm.
Dueling DDQN
This post documents my implementation of the Dueling Double Deep Q Network (Dueling DDQN) algorithm.
DDQN
This post documents my implementation of the Double Deep Q Network (DDQN) algorithm.
DQN
This post documents my implementation of the Deep Q Network (DQN) algorithm.