79 - HPC Café on February 11, 2025: Handling many small files and managing AI data sets/ClipID:56285 previous clip next clip

The automatic subtitles generated using Whisper Open AI in this video player (and in the Multistream video player) are provided for convenience and accessibility purposes. However, please note that accuracy and interpretation may vary. For more information, please refer to the FAQs (Paragraph 14).
Recording date 2025-02-11

Via

Free

Language

English

Organisational Unit

Zentrum für Nationales Hochleistungsrechnen Erlangen (NHR@FAU)

Producer

Zentrum für Nationales Hochleistungsrechnen Erlangen (NHR@FAU)

Speaker: Dr. Anna Kahler, NHR@FAU

Slides: https://hpc.fau.de/files/2025/02/HPC-Cafe-Small-Files-AI-DataSets-Feb-11-2025.pdf

Abstract:

We invite you to join us for a discussion on data handling, including the possibility of NHR@FAU providing access to popular data sets. As part of this discussion, we will present an overview of the various file systems available for data storage at NHR@FAU, covering key topics such as data archive formats, data copying, archiving, compressing, and unpacking, as well as recommendations for the most effective programs to use. Additionally, we will share best practices for efficient data storage and access in your SLURM scripts.

By taking a few simple steps, many common data handling issues can be resolved, which is crucial given that NHR@FAU supports over 1,000 users and inefficient data usage can impact not only individual workflows but also those of colleagues. Despite the importance of this issue, we continue to observe inefficient data handling practices and believe it is essential to revisit this topic time and again.

Material from past events is available at: https://hpc.fau.de/teaching/hpc-cafe/

More clips in this category "Friedrich-Alexander-Universität Erlangen-Nürnberg Zentralbereich"

2025-03-20
IdM-login
protected  
2025-02-24
IdM-login / Passwort
protected  
2025-02-24
IdM-login / Passwort
protected  
2025-02-24
IdM-login / Passwort
protected  
2025-02-24
IdM-login / Passwort
protected  
2025-02-24
IdM-login / Passwort
protected