Hello and welcome to my talk about using containers on the HPC system at NHR at FAU.

My name is Lukas Schröder and I'm part of the BayernKar E-Team here at the NHR.

This is the agenda.

At first I want to introduce you.

So why do you want to use containers and what are the upsides?

Then some container basics.

What are they actually?

How does it work in the background?

Then the most important part of course, how can you set up your own environment using

containers?

So starting with a fresh container or utilizing existing containers, especially you might know

Docker is the most used one or the most famous one.

How you can, for example, use them also on our systems for another software.

And then at the end some troubleshooting and best practices.

And of course, it's time for you to ask some questions.

First the introduction.

Why the first argument is you have portability and reproducibility.

So first one main goal is you can bring your own environment.

So inside a container you can set up something like an OS or I'll explain it closer.

So you have way more freedom inside.

And on the other hand, you also have then less dependence on software that is present

on clusters that we have.

Additionally, it is then packed as one file and this file can be easily copied and also

works on other systems that use, for example, app trainers.

This means the distribution is also easier.

And on the other hand, you have, as already said, an increased flexibility.

So system packages you cannot install on our clusters by your own.

You have to ask the admins.

You can install inside the container.

You can also get a feature that comes close to root access called fake root.

And also, as said, you can build it on your own device at home, test it there and then

use now the cluster hardware and copy it to our clusters and run it there or copy it to

other clusters.

So one example of why this is now also a current issue on our systems is that a lot of especially

AI people use Python PIP packages and to use them you use virtual environments usually

or conda environments.

And these create many, many small files.

These small files slow down the system in general because when you load many small files,

this takes way longer and takes down the performance of the storage system.

So what would be a solution?

So if you were there for the HPC Cafe on February, where it was about handling many small files,

you might remember one sentence from there.

Would you rather carry 1000 puzzle pieces with or without a box?

And containers, the container approach is to build a box for your environment.

So you build one big container where all your environment is set up.

This is just not Python files.

You can also increase it to other libraries, programs and so on.

And these files can be then loaded very fast because it's one big file.

And then the next motivation is because you should.

We and our systems encounter increasing problems with Python conflicts itself.