Welcome back to deep learning and today we want to talk about the final part of the architectures

and in particular we want to look into learning architectures.

Okay part five learning architectures.

Well the idea here is that we want to have self-developing network structures and they can be

optimized with respect to accuracy, floating point operations and of course you could simply do that

with a grid search but typically that's too time consuming. So there have been a couple of approaches

to do that and one of the ideas here in reference 22 is using reinforcement learning. So you train

a recurrent neural network to generate model descriptions of networks and you train this RNN

with reinforcement learning to maximize the expected accuracy. Of course there's also many

other options you can do reinforcement learning for small building blocks transferred to large

CNNs, genetic algorithms, energy based and there's actually plenty of ideas that you could follow

but they are all very expensive in terms of training time and if you want to look into those

approaches you really have to have a large cluster because otherwise you aren't able to actually

perform the experiments. So there's actually not too many groups in the world that are able to

do such kind of research right now. So you can see that also here many elements that we've seen

earlier pop up again there's the separable convolutions and many other things that you

can see here in the left hand side you see this normal cell which kind of looks like an inception

module if you look at the right hand side it kind of looks like later versions of the inception

modules where you have these separations and they are somehow concatenated and also use residual

connections and this somehow has been determined only by architecture search. Performance for

these on par with the squeeze and excitation networks with lower computational costs and

yeah there's also of course optimization possible for different size for example for mobile platforms.

ImageNet, where are we? Well we see that the ImageNet classification has dropped now below

five percent in most of the submissions. Substantial and significant improvements are more and more

difficult to show on this data set and the last official challenge was on CVPR in 2017 it's now

continued on Kaggle. There is new data sets that is being generated and is needed for example 3D

scenes human level understanding and those data sets are currently being generated there's for

example things like the MS-Coco data set or the Visual Genome data set which have replaced ImageNet

as state-of-the-art data set. Of course there's also different research directions like speed and

size of networks for mobile applications and in these situations ImageNet may still be a suitable

challenge. So let's come to some conclusions we see that the one by one filters to reduce the

parameters and add regularization is a very common technique. Inception modules are really nice

because they allow you to find the right balance between convolution and pooling. The residual

connections are a recipe that have been used over and over again and we've also seen that some of

the new architectures can actually be learned. So we see that there is a rise of deeper models

from five layers to more than a thousand. However often a smaller net is sufficient.

Of course this depends on the amount of training data you can only train those really big networks

if you have sufficient data and we've seen that sometimes it also makes sense to build wider layers

instead of deep layers. You remember we've already seen that in the universal approximation theorem

if we had infinitely wide layers then maybe we could fit everything into a single layer.

Okay so that brings us already to the outlook on the next couple of videos and what we want to

talk about is recurrent neural networks. We will look into long short-term memory cells,

we will look into truncated back propagation through time which is a key element in order

to be able to train those recurrent networks and we finally have a look at the long short-term

memory cell one of the key ideas that have been driven by Schmidhuber and Hochreiter.

Another idea that came up by Cho are the gated recurrent units which can somehow be a bridge

between the traditional recurrent cells and the long short-term memory cells. Well let's look at

some comprehensive questions. So what are the advantages of deeper models in comparison to

shallow networks? Why can we say that residual networks learn an ensemble of shallow networks?

You remember I hinted on that slide this is a very important one if you want to prepare for the exam