Welcome back to deep learning and today we want to talk about deep reinforcement learning.

So I have a couple of slides for you and of course we want to build on the concepts that

we've seen in reinforcement learning but we talk about deep Q learning today.

And one of the very well known examples is the human control through deep reinforcement

learning here in reference four.

This was done by Google DeepMind and they showed a neural network is able to play Atari

games.

So the idea here is to directly learn the action value function using a deep network

and the inputs are essentially the three subsequent video frames from the game and this is processed

by a deep network and it produces the best next action.

So the idea is now to use this deep reinforcement framework to learn the best next controller

movements and they do convolutional layers for the frame processing and then fully connected

layers for the final decision making and here you see the main idea of the architecture.

So there's these convolutional layers, relu's, you have the input frames that are processed

by these then you go into fully connected layers and again fully connected layers and

then you produce directly the output and you can see that in Atari games this is a very

limited set so you can either do no action then there is essentially eight directions,

there's a fire button and there's the eight directions plus the fire button.

So that's all of the different things that you can do so it's a limited domain and you

can then train your system with that.

Well it's a deep queue network that directly applies queue learning.

The state of the game is essentially the current plus three previous frames as an image stack

so you have a rather fuzzy way of incorporating memory and state.

Then you have 18 outputs that are associated with the different actions and each output

estimates the action for the given input.

You don't have a label and a cost function but you update with respect to maximize the

future reward.

There's a reward of plus one when the game score is increased and a reward of minus one

when the game score is decreased otherwise it's zero.

They use an epsilon greedy policy with epsilon decreasing to a low value during the training

and they use a semi-gradient form of the queue learning to update the network weights w and

again they use mini-batches to accumulate the weight updates.

So they have this target network and it's updated using the following rule so you can

see that this is very close to what we have seen in the previous video so again you have

the weights and you update them with respect to the rewards.

Now the problem is of course that this gamma and selection of the maximum queue function

is a function of the weights again.

So you somehow have a dependency of the maximization in there and the weights you're trying to

update.

So your target changes simultaneously with the weights that we want to learn and this

can actually lead to oscillations or divergence of your weights.

So this is not very good.

So they introduce a second target network and this after C steps they generate this

by copying the weights of the action value network to a duplicate network and keep them

fixed.

So you use the output Q bar of the target network as a target to stabilize the previous

maximization.

So you don't use Q hat the function that you're trying to learn but you use the Q bar which

is the kind of fixed version that you use for a couple of iterations.

Another trick they have been using is experience replay and here the idea is to reduce the