Welcome back to deep learning and today we want to talk about the final part of the architectures

and in particular we want to look into learning architectures.

Okay part five learning architectures.

Well the idea here is that we want to have self-developing network structures and they can be

optimized with respect to accuracy, floating point operations and of course you could simply do that

with a grid search but typically that's too time consuming. So there have been a couple of approaches

to do that and one of the ideas here in reference 22 is using reinforcement learning. So you train

a recurrent neural network to generate model descriptions of networks and you train this RNN

with reinforcement learning to maximize the expected accuracy. Of course there's also many

other options you can do reinforcement learning for small building blocks transferred to large

CNNs, genetic algorithms, energy based and there's actually plenty of ideas that you could follow

but they are all very expensive in terms of training time and if you want to look into those

approaches you really have to have a large cluster because otherwise you aren't able to actually

perform the experiments. So there's actually not too many groups in the world that are able to

do such kind of research right now. So you can see that also here many elements that we've seen

earlier pop up again there's the separable convolutions and many other things that you

can see here in the left hand side you see this normal cell which kind of looks like an inception

module. If you look at the right hand side it kind of looks like later versions of the inception

modules where you have these separations and they are somehow concatenated and also use residual

connections and this somehow has been determined only by architecture search. Performance for

ImageNet is on par with the squeeze and excitation networks with lower computational costs and

yeah there's also of course optimization possible for different size for example for mobile platforms.

ImageNet where are we? Well we see that the ImageNet classification has dropped now below

five percent in most of the submissions. Substantial and significant improvements are more

and more difficult to show on this data set and the last official challenge was on CVPR in 2017.

It's now continued on Kaggle. There is new data sets that is being generated and is needed for

example 3D scenes, human level understanding and those data sets are currently being generated.

There's for example things like the MS-Coco data set or the Visual Genome data set which have

replaced ImageNet as state-of-the-art data set. Of course there's also different research directions

like speed and size of networks for mobile applications and in these situations ImageNet

may still be a suitable challenge. So let's come to some conclusions. We see that the

one-by-one filters to reduce the parameters and add regularization is a very common technique.

Inception modules are really nice because they allow you to find the right balance between

convolution and pooling. The residual connections are a recipe that have been used over and over

again and we've also seen that some of the new architectures can actually be learned.

So we see that there is a rise of deeper models from five layers to more than a thousand. However,

often a smaller net is sufficient. Of course this depends on the amount of training data.

You can only train those really big networks if you have sufficient data and we've seen that

sometimes it also makes sense to build wider layers instead of deep layers. You remember

we've already seen that in the universal approximation theorem. If we had infinitely

wide layers then maybe we could fit everything into a single layer. Okay so that brings us

already to the outlook on the next couple of videos and what we want to talk about is recurrent

neural networks. We will look into long short-term memory cells, we will look into truncated back

propagation through time which is a key element in order to be able to train those recurrent

networks and we finally have a look at the long short-term memory cell, one of the key ideas

that have been driven by Schmidt-Huber and Hochreiter. Another idea that came up by Cho

are the gated recurrent units which can somehow be a bridge between the traditional recurrent cells

and the long short-term memory cells. Well let's look at some comprehensive questions.

So what are the advantages of deeper models in comparison to shallow networks? Why can we say

that residual networks learn an ensemble of shallow networks? You remember I hinted on that slide