Welcome everybody to deep learning. So today we want to conclude talking about different

regularization methods. We want to talk in particular about one more technique that is

called multitask learning. In multitask learning we want to extend the previous concepts. Previously

we only had one network for one task. Then we had transfer learning to reuse the network.

The question is can we do more? Can we do it in a better way? There are some real world

examples. For example if you learn to play the piano and the violin then in both tasks

you require a good hearing, a sense of rhythm, music notation and so on. So there are some

things that can be shared. Also soccer and basketball training both require stamina,

speed, body awareness and body-eye coordination. So if you learn the one then you typically

also have benefits for the other. So this would be even better than reusing as you learn

simultaneously and then provide a better understanding of the shared underlying concepts. The idea

is now that we train our network simultaneously on multiple related tasks. So we adapt the

loss function to assess performance for multiple tasks. And this then results in a multitask

learning that introduces a so-called inductive bias. As a result we prefer a model that can

explain more than a single task. Also this reduces the risk of overfitting on one particular

task and our model generalizes better. So let's look at the setup. So we have some

shared input layers and these are like the feature extraction layers or representation

layers. Then we split at some point where we go into the task specific layers to evaluate

on task A, task B and task C. They may be very different but somehow related because

otherwise it wouldn't make sense to share the previous layers. So several hidden layers

are shared between all of the tasks. As already shown by Baxter in reference to multitask

learning of N tasks reduces the chance of overfitting by an order of N. Instead of hard

sharing you can also do soft parameter sharing. It introduces an additional loss. So you constrain

the activations in the respective layers to be similar. Hence each model has its own parameters

but we link them together to perform similar yet different extraction steps in the constraint

layers. You can do that for example with an L2 norm or other norms that make them similar.

Now we still have to talk about auxiliary tasks. All of these tasks should have their

own purpose. You may also just include auxiliary tasks just because you want to create a more

stable network. So one example here is facial landmark detection by Zang in reference 22.

They essentially want to detect facial landmarks but it is impeded by occlusions and post variances.

So they start simultaneously to learn landmarks and subtly related tasks like the face pose,

smiling versus not smiling, glasses versus no glasses and gender. They had this information

available and then you can set this up in a multitask learning framework as you see

here in the network architecture. In the results you see how the auxiliary tasks support the

detection of facial landmarks. They compare this to a CNN, a cascaded CNN and a multitask

network with the auxiliary tasks. The landmark detection is improved by the introduction

of these auxiliary tasks. So certain features may be difficult to learn for one task but

may be easier for a related one. The auxiliary task can help to steer the training in a specific

direction and we somehow include prior knowledge by choosing appropriate auxiliary tasks. Of

course tasks can have different convergence rates. To alleviate the problem you can then

also introduce task based early stopping. An open research question is what tasks are

appropriate auxiliary tasks. So this is something we cannot generally answer but it is typically

determined by experimental validation. So next time on deep learning we start with a

new topic where we look into some practical recommendations to actually make things work.

So with all what you've seen already we are pretty far but there's a couple of hints that

will make your life easier so I definitely recommend watching the next couple of lectures.

We will look into how to evaluate performance and deal with the most common problems that

essentially everybody has to face in the beginning. Also we look at concrete case studies with

all the pieces that you've learned until now. I have a couple of comprehensive questions

like what is the bias variance trade-off, what is model capacity, describe three techniques