This audio clip is presented by the University of Erlangen-Nürnberg.

So, since these long short term memory networks are so important, let me just remind you briefly of what we said.

The idea is that the network will contain some memory cells, but that whether they are being deleted or written or read is determined by other neurons, and that most of the time really nothing happens.

So then, for example, these other neurons can be used to detect and detect the same memory cells as the other neurons.

So, this is a very interesting example of how we can do that.

So, let's start with the first example.

So then, for example, these other neurons can produce the value of a so-called forget gate neuron,

and depending on the forget gate, well, you forget the content of the memory cell or you keep it.

You can also write something new into the memory cell.

So, you have an input gate that determines whether a potential new value will be added to the previous value of the memory cell.

And so, by combining a forget operation with such a write operation, you replace the contents of the memory cell.

And then finally, of course, there must be a read operation.

So, again, there will be a gate neuron that determines whether such a read operation takes place.

Here it's called the output gate, and if this is one, then you take out the content of the memory cell

and you feed it further into the network, say into higher layers of the network where you do something with it.

So, these are the important operations, delete, write, and read.

As a matter of fact, it's also convenient to make all these operations not only dependent on the current input

to the network or the input to that layer, but also dependent on the previous output

and the previous time step of this memory layer, and that's contained in here.

So, what can happen then is you trigger something, you read out something, and depending on what you read out

from the memory cell, in the next step, say, another memory neuron will be read out, and so on and so on.

So, you recall whole sequences.

Okay, so then I told you how to program this in Keras in principle, and then we discussed two examples

of such a memory network, how such a recurrent network can work.

This one was relatively simple.

You write a value into the network, and at some later time, you give it a recall signal,

and then it should reproduce this value.

And so, one way to implement this would be to have three input neurons.

One of the input neurons has the function that if it is set to one, then that signals to the network

that now it should pay attention to the second neuron, because that contains a value that is to be stored

inside the memory.

And then, at a later time, which can be random, at yet another neuron, there will be a flag.

There will be a one telling it now is time to recall, and then we hope that the network outputs

the previously stored value.

So, the way to think about these recurrent neural networks is always you have a time sequence of inputs

that could be several neurons, like here, and you also have a time sequence of outputs,

could be several neurons or only one neuron, like here, and then you should make yourself a picture

like this just to understand what you would like to be the ideal output,

because in order to train the network, you have to create a batch of input sequences

together with the desired output sequences, and that is what you train the network on.

So, that was a network that can recall a number.

We also looked at a network that counts down.

So, you tell it at one particular point a number that is an integer, which has the meaning that

after this number of steps, the network should output some signal, like a one, for example.

So, the typical combination of input time sequences and output time sequences would look like the one shown here.

And again, we managed to train such a network.

Okay, and so here, for example, you saw time traces, time running vertical,

and the learning episodes where you learn on batches,

and then you see that initially the network doesn't understand what to do,

but finally, indeed, after a delay of five time steps in this example, it outputs the signal.