Welcome everybody to deep learning. So today we want to conclude talking about different

regularization methods and we want to talk in particular about one more technique that

is called multitask learning. Also because I'm lazy, so you know. Multitask learning,

we want to extend the previous concept. So previously we only had one network for one

task and then we had transfer learning to reuse the network.

So that is transfer learning and it has been done in principle for many decades.

But the question is can we do more? Can we do it in a better way? And there are some

real world examples. For example, if you learn to play the piano and the violin, then in

both tasks you require good hearing, sense of rhythm, music notation and so on. So there

are some things that can be shared. Or also soccer and basketball training. Both require

stamina, speed, body awareness, body-eye coordination. So if you learn the one, then you typically

also have benefits for the other.

We're not starting from scratch because we've been doing this with humans for millennia.

So this would be even better than reusing. So you learn simultaneously and then provide

a better understanding of the shared underlying concepts.

So the idea now is that we train and network simultaneously on multiple related tasks.

So we adapt the loss function to assess performance for multiple tasks. And this then gives a

multitask learning that introduces a so-called inductive bias. We prefer a model that can

explain more than a single task.

The stuff that works best is really simple.

Also this reduces the risk of overfitting on one particular task and our model generalizes

better. So let's look at the setup. So we have some shared input layers. So these are

like the feature extraction layers and the representation layers. And then we split at

some point where we go into task specific layers and then evaluate it on task A, task

B, task C. And they may be very different but somehow related because otherwise it would

make sense to share the previous layers.

So several hidden layers are shared between all of the tasks. And as already shown by

Baxter in 97, multitask learning of n tasks reduces the chance of overfitting by an order

of n.

There is nothing more practical than a good theory.

Instead of hard sharing you can also do soft parameter sharing. Soft parameter sharing

would now introduce an additional loss. So you constrain the activations in the particular

layers to be similar. So each model has its own parameters but we somehow link them together

to perform similar extraction steps yet different extraction steps in the constrained layers.

And you can do that for example with an L2 norm or other norms that make them similar.

Now we still have to talk about the auxiliary tasks. So all of these tasks should have an

own purpose. You may also just include auxiliary tasks just because you want to create a more

stable network. So one example here is facial landmark detection by Zang. And they essentially

want to detect facial landmarks but this is impeded by occlusion and pose variances.

So they start simultaneously to learn landmarks and subtly related tasks like the face pose,

smiling, not smiling, glasses, no glasses, occlusion and gender. So they had this information

available and then you can set this up in a multitask learning framework as you see

here in the network architecture. And in the results you see that they then have the auxiliary

tasks here but they're actually interested in the facial landmarks and they compare this

to a CNN, a cascaded CNN and now they're multitask network with the auxiliary tasks and they

can show that also the landmark detection is improved by introduction of these auxiliary

tasks. So certain features may be difficult to learn for one task but it may be easier

for a related one. So the auxiliary tasks can help to steer the training in a specific

direction and we somehow include prior knowledge by choosing appropriate auxiliary tasks. And

of course then tasks can have different convergence rates so you can then also introduce task-based