Welcome back to deep learning to the last video where we discussed the different algorithms

regarding generative adversarial networks and today we want to look into the fifth part

of our lecture and these are essentially more tricks of the trade concerning GANs.

One trick that can help you quite a bit is one-sided label smoothing. So what you may want to do is

replace your targets of the real samples with a smoothed version. So instead of using a one

probability you use a 0.9 probability but you do not use the same for the fake samples. So you

don't change their label to zero because otherwise this will reinforce incorrect behavior. So your

generator would produce samples that resemble the data or samples it already makes. Benefits are

that you can prevent the discriminator from giving very large gradients to your generator and you

also prevent extrapolating to encourage extreme samples. Is balancing between the generator and

the discriminator necessary? No it's not. The GANs work by estimating the ratio of data model density

so the ratio is estimated correctly only when the discriminator is optimal so it's fine if your

discriminator overpowers the generator. When the discriminator gets too good your gradients of

course may vanish then you can use tricks like the non-saturating laws the Wasserstein GANs as

we talked about earlier and you may also run into the problem that your generator's gradients may

get too large and in this case you can use the trick of label smoothing. Of course you can also

work with deep convolutional GANs so this is the C-GAN where you implement a deep learning approach

into the generator so you can replace pooling layers with stride convolutions and transposed

convolutions. You can fully remove the connected hidden layers for deeper architectures and the

generator then typically uses ReLU activations except for the output layer in which you use

a tungent superpolychos and the discriminator for example here uses a leaky ReLU activation for

all the layers and they use batch normalization and if you do that then you may end up in the

following problem you can see here some generation results and within the batches there may be a very

strong intra-batch correlation so within the batch all of the generated images look very similar.

And this brings us to the concept of virtual batch normalization so you don't want to use

one batch normalization instance for both mini-batches. You could use two separate batch

normalizations or even better you use the virtual batch normalization and in case this is too

expensive you choose instance normalization for each sample and subtract the mean and divide by

the standard deviation. In case you choose virtual batch normalization then you create a reference

batch R of random samples and fix them once at the start of the training and then for each Xi of the

current mini-batch you create a new virtual batch that is the reference batch union d Xi and then

you compute the mean and standard deviation of this virtual batch and you always need to propagate

then R forward in addition to the current batch. This then allows you to normalize Xi with these

statistics so this may be kind of expensive but we have seen that this is very useful for stabilizing

the training and remove the intra-batch correlations. There's also the idea of

historical averaging so there you add a penalty term that punishes weights which are rather far

away from the historical average and this historical average of the parameters can then be updated in an

online fashion. Similar tricks from reinforcement learning can also work for generative adversarial

networks like experience replay. You keep a replay buffer of past generations and occasionally show

them and you keep checkpoints from the past generator and discriminator and occasionally swap

them out for a few iterations. So if you do so then you can do things like the dc-gun. Here are

bedrooms after just one epoch and you can see that you are able to generate quite a few different

bedrooms. So very interesting what kind of diversity in terms of generation you can actually achieve.

Another interesting observation is that you can do vector arithmetic on the generated images so

you can generate for example the mean of three instances of man with glasses and with this mean

then you can subtract for example the mean of man without glasses and then you compute the mean

of woman without glasses and add it on top and what you get is woman with glasses. So you can

really use the constrained generation with this trick in order to generate something where you

potentially don't have a conditioning variable for. So the guns learn a distribution representation

that disentangles the concept of gender from the concept of wearing glasses and if you're interested