DKRZs JupyterHub

You like to analyse the big data sets stored at DKRZ by using Python, R or Julia scripts?

Then, DKRZs JupytherHub is the perfect tool to do so!

It allows to execute the notebook directly on the DKRZ HPC system Levante and therefore has the following advantages:

• no need to copy any data, e.g. to your home device

• Levantes nodes which might be faster than your ones

• you can make use of parallel computation of your scripts

• project compute resources can be used

• nextGEMS and DYAMOND provide intake catalogues of their experiments, so you can search and load data files directly into your Python script

• notebooks can easily be shared with others

Note

JupyterHub is available at https://jupyterhub.dkrz.de. You need a DKRZ user account (user-id), which has to be part of a compute project with resources to Levante.

DKRZs Documentations pages

• DKRZ JupyterHub documentation (docs.dkrz.de)

• DKRZ JupyterHub blog posts (docs.dkrz.de)

Our related tutorials, tips and tricks

First step tutorials

From our ICON beginners guide (Python) DYAMOND Winter dpp0029/dpp0033 - a beginner’s guide:

• detailed explenaition how to setup a Jupyter notebook

• Example notebook

Searching for data files and directly load it into your Python script:

• Loading data from the catalog

• or use Getting file names from intake with a command-line python script to get detailed file infos or use it within CDO post-processing

Query the memory footprint of your Python jobs

To query the memory footprint of your (processing) jobs, you can use these commands:

# put these function definitions in your .bashrc
function maxmem_running() {  sstat --format=AveCPU,MaxRSS,AveRSS,JobID -j $1 --allsteps; }
function maxmem_terminated() {  sacct --format=JobID,JobName,MaxRSS,Elapsed -j $1 ; }

# call on a job-id
maxmem_running MY_RUNNING_JOB_ID
maxmem_terminated MY_TERMINATED_JOB_ID