We present a new learning-based method for denoising of path-traced images from volumetric

medical data, including a specialized loss function, features tailored for this purpose

and a novel network architecture.

Path-traced videos of CT scans are of high value for surgery planning, medical education

and the like.

As convergence requires tracing thousands of lighting paths, fast previews with only a

few light samples exhibit a high amount of noise.

We present a learning-based method to denoise these noisy images using a convolutional neural

network.

To gain as much speedup as possible, we use images of only one sample per pixel as an

input, which unfortunately do not contain enough information for reconstruction.

Apart from the noisy one sample per pixel image itself, we therefore additionally feed

several further features to our network, like it is done for the surface-only case.

In the volumetric case, however, finding features is not trivial, as properties like depth and

normal are not well defined.

We opt for seven additional features, split into primary and secondary ones.

The primary features stem from the raceway up to the first scatter event.

The first one is the position of the first scatter event, interpreted as the position

of the pixel.

The second one, the gradient direction of this position, as an approximate normal vector.

The third primary feature is the albedo of the first scatter event.

The fourth feature texture combines multiple volumetric characteristics, gradient magnitude,

accumulated opacity and density of the volume, color-coded in R, G and B.

The secondary features deal with the further course of the wave.

Here we utilize the color of the first surface scatter event, the albedo at the second scatter

position and volumetric characteristics of the wave from the first to the second scatter

event.

Note that the features, just like the one sample per pixel color input, suffer from

noise due to undersampling, especially severe in the secondary features.

Our basic network is a standard unit with skip connections.

However, to be able to gainfully include the very noisy secondary features, we duplicate

it and create dual network, where one part computes direct effects using the primary

features and the other one computes indirect effects using the secondary features.

For evaluating the output of this network, we furthermore utilize a combination of a

static loss term and a relativistic discriminator to further sharpen the image.

Moreover, we include the temporal domain by taking the preceding and subsequent frame

into account.

To be able to do so, we reproject both the color input and all features to the current

frame.

The triplets are preprocessed in a number of separate autoencoders, which complete our

network architecture.

One can clearly see the improvements in the video results when adding the different components

one after another.

Note the gain in temporal stability when the reprojection stage is added to the network.

Our method creates stunning results, correctly reproducing both surface-like hard materials

with their typical specular highlights and fuzzy volumetric regions showing a high amount

of transparency.

We compare our method to the optics denoiser applied to the one sample per pixel color

inputs.

Our approach clearly outperforms it, generating images very similar to the target image in