The following content has been provided by the University of Erlangen-Nürnberg.

Hello everybody. First of all I would like to introduce myself. My name is Thomas Köhler and today I have the pleasure to give you a lecture about a topic that is also closely related to my PhD project.

And today we will talk about super resolution image reconstruction.

So first of all this is the outline of this lecture and last time you already heard what is super resolution and you heard something about cameras and sampling in general.

Today we discussed the sampling theorem which is important if we sample continuous functions. We saw how the sampling of real camera works.

We saw quantization and image noise and the goal of this or the result of the sampling of real cameras is a degraded idle image.

Now we don't have idle images after the sampling but we have images which are affected by image noise for example, by blur caused by the point spread function of a camera and we have a smaller resolution of our images.

So today we will discuss methods to enhance the resolution of such images and in the first instance it sounds like magic. How can you enhance the resolution of a low resolution image?

Maybe this is impossible because we cannot bring back information that is lost but all we need to get a good estimation for our idle image before the imaging process is a bit mass especially probability theory and numerical optimization techniques.

And here we discuss single and multi frame super resolution and I will present you statistically paced super resolution methods based on multi frame super resolution.

And here we will see two methods. The first one is maximum likelihood estimation so we can utilize maximum likelihood approach to estimate a high resolution image.

And the second method is an extension of maximum likelihood called maximum a posteriori estimation.

Okay so first of all we will start with a general introduction in single and multi frame super resolution.

So the first group of methods is single frame super resolution and maybe some guys say okay single frame super resolution is no super resolution it's just interpolation but in literature a lot of authors also talk about super resolution if they utilize only single images.

So what is the goal of single frame super resolution so in this case we have one single low resolution image. So we have no frame sequence just one image and this image is degraded by noise by blur and by a low spatial resolution.

And now we would like to enhance the resolution of this image and there are different kind of methods available.

I don't want to go into detail here we have first of all learning based methods.

Here we try to estimate a high resolution image using training data so we would like to learn the image process of the camera based on some given training samples.

A different kind of approach is frequency interpolation. Here we represent an image in the frequency domain for example we can use wavelet coefficients.

And now we would like to recover the high frequency information that is not present in the low resolution images.

So if you remember on the sampling theory we have this cutoff frequency and in our low resolution observation we lost high frequency information.

And this frequency interpolation methods recover exactly this high frequency information.

So this is just a brief overview of single frame methods. The main topic in this lecture is multi frame super resolution and here on this slide you see the basic principle of this methods.

So in this case in the case of multi frame super resolution we have now sequences of images available so we have no longer only a single image but multiple images.

And the basic principle is shown here on the right hand side in this figure and the idea is that we capture a sequence of frames.

So the blue, the red and the green one which is shifted to each other and because we have here shifts in between the pixels.

So for example we have this blue pixels from the first image shifted to the red and the green positions and the second and the third frame we can sample the space in between low resolution image more precise.

It is important to note that we need of course sub pixel shifts here to sample the space in between two pixels.

This sub pixel shift can be caused by a moving camera of course so if you move your camera to different positions in space to different view points you can capture a sequence of slightly shifted images.

A different kind of motion is object motion this is equivalent of course if you have a fixed camera and you capture an object or a scene that is moving this is the same like a moving camera.

So now we have for multi frame super resolution such a sequence of frames so the pixels from the blue pixels, the green and the red pixels and our goal is now to reconstruct a high resolution ideal image from our low resolution frames.

So we would like to recover the information in between our low resolution pixels.

Okay first of all I would like to introduce the basic notation of multi frame super resolution since this is not very common.

Okay so first of all if we would like to denote an image A we can of course use the well known matrix notation so we have an image A which is a matrix A11, A12, A13 and so on.

This is a matrix notation and we have here real values.

Now we can denote a linearized image A and here we have a vector A which we can create from our matrix via line scanning and a linearized image is now a vector A11, A12, A10 and so on.

This is the same as A where we concatenate all pixels into one large one dimensional vector A.

In the following we will denote high resolution images as X and a high resolution image as an N dimensional vector.

Low resolution images are denoted as Y and these images are M dimensional.

And we deal here with sequences of low resolution frames and we denote the frame index as this index here.

So we have a Y of K which is the K frame out of a low resolution image sequence.

So first of all if we would like to perform multi frame super resolution we have to think about a generative image model.

A generative image model describes how an ideal image is mapped to a real image of our camera and there are different artifacts introduced in the model.

And the starting point here is to derive this model as an ideal image which is shown here on the left hand side.

If you have a look at this ideal image you have almost perfect edges.

So you have no blur, you have a high resolution and you have no noise visible.

Afterwards the first effect of our model is a geometric warp which means that we can shift and rotate for example our ideal image to a second image.

This is shown here. This is a rotated and translated version of the first one.

Afterwards we have to blur our warped image. This is shown here.

And if you compare this image with this image you see that the edges here are no longer sharp and we lose high resolution, high frequency information here at the edges.