Welcome back to deep learning. So today I want to talk to you about ideas how we can reuse prior knowledge and integrate it into deep networks.

And this is actually something that we've been doing in a large research project that is being funded by the European Research Council.

And I thought these ideas are also interesting for you, so I decided to include them in the lecture.

So this brings us to the topic of known operator learning.

And known operator learning is a very different approach because we try to reuse knowledge that we already have about the problem and therefore have to learn fewer parameters.

This is very much in contrast to what we know from traditional deep learning. If you say so, traditional deep learning, there we often try to learn everything from the data.

Now, the reason why we want to learn everything from the data is, of course, because we know very little about how the network is actually supposed to look like.

So we try to learn everything from the data in order to get rid of problems.

And in particular, this is the case for perceptual tasks where we have very little knowledge how humans actually solve the task.

So the human brain for us is largely a black box, and we try to find a matching black box that is also solving this problem.

So I brought this example here from Florin Gessu. And you remember I showed this already in the introduction.

So here we had this kind of reinforcement learning type of approach where we then motivate our search for organs in the body by reinforcement learning.

So we look at small patches in the image and then decide where to move in the next step in order to approach the specific landmark.

So we kind of can introduce here how we interpret the image or how an radiologist interprets the image and how he would move towards a certain landmark.

And of course, we had this multiscale approach and so on.

The main reason why we do it in this way is, of course, because we don't know how the actual brain works and what the radiologist is actually thinking.

But we can at least mimic somehow his work style and the way how we approach this here.

Well, but this is generally not the case for all problems.

And deep learning is so popular that it's being applied to many, many different problems other than perceptual tasks.

So, for example, people have been using this in order to model CT reconstruction.

So here the problem is that you have a set of projection data shown here on the left and you want to reconstruct slice data shown on the right hand side.

So the problem is very well researched on. We know solutions to this problem already since 1917.

But there's, of course, problems with artifacts and image quality and so on, dynamics, which make the problem hard.

And therefore we would like to find improved reconstruction methods.

One problem, for example, is called the limited angle problem.

So if we only rotate by, let's say, 120 degrees instead of a full rotation, you get slice images like shown here on the left hand side.

So they are full of artifact and you can barely see what is shown on the image.

We have the matching slice image on the right hand side.

And if you look at the right hand side image, you can see this is a cut through the torso.

It shows the lungs. It shows the heart. It shows the spine and the ribs in the front.

We barely see the ribs and the spine in the image on the left.

But we have methods that can do image to image completion.

We've seen that we can even use this for inpainting to interpolate missing information in images.

So why not just apply it to complete the reconstruction?

And this has actually been done. And I can show you one result here.

So this actually works. This is done for an unseen person.

So this has been trained with slices from 10 other persons and evaluated here on the 11th one.

So this person has never been seen. And you can see it very nicely reconstructs the ribs, the torso, the chest wall is there that is barely visible in the input image.

And we can also see a very nice appearance here. So this is pretty cool.

But to be honest, this is a medical image. People do diagnosis on this.

So let's put it a bit to the test and hide a lesion.

So we put it here exactly in the chest wall.

And this is kind of mean because this is exactly where we had the worst image quality.

I'm also showing a blow up view on the bottom right.

So you see that the lesion is there and it has considerably higher contrast than the surrounding tissue.

Now, if I show you this, you can see the input that we would show to our unit.

So you can see the lesion is barely visible in the blow up view.

You can actually see it, but it has a lot of artifact.

And now the question is, will it be preserved or will it be removed from the image?

And well, it's there. You can see the lesion is here. So that's pretty cool.