Welcome back to deep learning. So today we finally want to look into degenerative adversarial networks, which is a key technology in unsupervised deep learning.

So let's see what I have for you here. Well, the unsupervised deep learning part 3, generative adversarial networks, GANs.

The key idea of GANs is to play the following game. You have a generator and a discriminator.

Now the generator, you could argue, is somebody who generates a fake image.

And then the discriminator has to figure out whether the generator actually produced something that's real or something which is fake.

So the discriminator can decide fake or real. And in order to train the discriminator, he has access to many real data observations.

So the outcome of the discriminator then is whether the input was real or fake.

Well, of course, this is difficult to ask persons and artists to draw things.

So we replace the two with deep neural networks. And we have D, that is the discriminator, and we have G, that is the generator.

And the generator receives some latent input, some noise variable Z.

And from the noise variable and the parameters, it produces some image and the discriminator then tries to figure out whether this was a real or fake image.

So the output of the discriminator is going to be one for real and zero for fake.

Once we have found this kind of neural network representation, then we are also able to describe a loss.

And the loss of our discriminator is to minimize the following function that is dependent on the parameters of the discriminator and the parameters of the generator.

And it is essentially minimizing the expected value of X from the data.

And this is simply the logarithm of the output of our discriminator for real samples minus the expected value of some generated noise.

And that is the logarithm of one minus the discriminator of the generator of some noise.

So it's trained to distinguish real data samples from fake ones.

Now, if you want to train the generator, you simply minimize the loss of the generator, that is the negative loss of the discriminator.

So the generator minimizes the log probability of the discriminator being correct.

So you train to generate domain images to fool D.

Optionally, you can run k steps of one player for every step of the other player.

And the equilibrium is a saddle point of the discriminator loss.

If you look into this in more detail, then you can find that the loss of the generator is directly tied to the negative loss of the discriminator.

So you can summarize this game with a value function specifying the discriminator's payoff that is given as V.

And this is the negative loss of the discriminator. And this then results in the following min-max game.

So the optimal parameter set of the generator can be determining by maximizing V with respect to the discriminator and nested into a minimization of the parameters of G with respect to the same value function.

So let's have a look at the optimal discriminator. And there is a key assumption that is both densities are non-zero everywhere.

Because otherwise, some input values would never be trained and the discriminator would have undetermined behavior in those areas.

Then you solve with respect to the gradient of the discriminator loss with respect to the discriminator to be zero.

And then you can find the optimal discriminator for any data distribution and any model distribution in the following way.

The optimal discriminator is the distribution of the data divided by the distribution of the data plus the distribution of the model over all your input domain of X.

Unfortunately, this optimal discriminator is theoretical and unachievable. So it's key for GANs to have an approximation mechanism.

And GANs use supervised learning to estimate this ratio. And then this leads to the problem of underfitting and overfitting.

Now what else can we do? We can do non-saturating games that we modify the generators loss. And then in this example, we are no longer using the same function for both.

But instead we have a new loss for the generator where we simply compute the expected value of the logarithm of the discriminator of the generator given some input noise.

In minmax, G minimizes the log probability of D being correct. In this solution, G minimizes the log probability of D being mistaken.

It's heuristically motivated because it fights the vanishing gradient of G when D is too smart. And this is particularly a problem in the beginning.

However, the equilibrium is no longer describable using a single loss.

So there's also things like extensions that are quite popular, like the feature matching loss or the perceptual loss.

Here then G is trying to match the expected value of features f of x of some intermediate layer of D.

And you've seen this already that f can be, for example, some other network and some layer 3 or layer 5 representation.

And then you want the expected values of these representations to be the same given for real inputs as well as for generated noise images.

So here you want to prevent the overtraining of the generator on the current discriminator.

And by the way, this is also a popular loss in many other domains. What else can be done?

Well, there's the so-called Wasserstein loss, and it's derived from the Wasserstein distance, which is also known as the Earth Movers distance.

And here you learn a discriminator that maximizes the discrepancy between the real and fake samples.

And at the same time, you restrict the gradient to stay beyond a certain limit.

So you essentially limit the gradient towards a specific Lipschitz constant, which is the maximum slope of the gradient.

And here in the image on the right hand side, you can see that out of the red discrimination curve, which saturates very quickly,