Hello, my name is Simon Grosche and I'm glad to present to you our latest work for MMSP

This presentation is about a novel end-to-end network for reconstruction of non-regularly

sampled image data using locally fully connected layers.

It is a joint work together with Fabian Brandt and André Cout from the Multimedia Communications

and Signal Processing Chair of Friedrich-Alexander-Universität of Erlangen in Nuremberg in Germany.

This work is built upon non-regular sampling.

Let me first introduce you to three different sampling sensors.

You see a reference image on the left.

With this reference image, we can think of several sampling sensors with half the resolution

in both dimensions, meaning a quarter of the number of pixels.

First, a low resolution sensor is shown here.

With this low resolution sensor, basically four pixels out of the reference image are

built into one measurement.

If you want to upscale this to the resolution of the reference image, you can use something

like bicubic upscaling or any super resolution method.

Next, we have two non-regular sampling sensors.

In the middle, the quarter sampling sensor is shown.

Here, instead of binning four pixels, only one out of four pixels is measured.

These measured pixels, however, are not placed regularly but non-regularly across the sensor.

This way, a non-regular sampling is achieved and one can show that this leads to a higher

quality of the reconstructed image.

Since the quarter sampling sensor has a lower light efficiency than the low resolution sensor,

meaning only a quarter of the light is coming in, the three-quarter sampling sensor was

proposed recently.

Here, the game is turned around a bit.

Instead of measuring one pixel, the other three pixels are integrated to one measurement.

Altogether, we need to sum up here that non-regular sampling enables a higher resolution per pixel

after appropriate reconstruction.

So far, the reconstruction methods have mostly been classical methods.

However, in this talk, we will show a new method that uses neural networks for the reconstruction

of quarter sampling data and three-quarter sampling data.

Let us first revisit the reconstruction using non-regular and using neural networks for

the different sensor layers that already exist.

On the left side, we see two existing methods.

The first method in A is the so-called very deep super resolution network, short VDSR.

Here, we can see that the low resolution sensor is first emulated.

Then, the measurements with the half of the resolution in both dimensions is fed into

a bicubic upscaling.

And the bicubically upscaled image is then fed into the VDSR network, consisting mainly

of convolutions and relu layers.

Additionally, there is a skip connection to the output, meaning only the residual is learned.

Based on this VDSR concept, we recently proposed a year ago a method called VDSRQS for non-regular

sampling data.

Here, a quarter sampling sensor is used in the beginning and the reference image is multiplied

with the mask of this quarter sampling sensor first.

The measurements are then reconstructed with a classical approach called FSR.

Now, the result of this initial reconstruction is then fed into a VDSR network.

There are some small adaptations, but please be referred to the paper if you are more interested

in this.