Hints for processing data

Tip

To query the memory footprint of your (processing) jobs, you can use these commands:

# put these function definitions in your .bashrc
function maxmem_running() {  sstat --format=AveCPU,MaxRSS,AveRSS,JobID -j $1 --allsteps; }
function maxmem_terminated() {  sacct --format=JobID,JobName,MaxRSS,Elapsed -j $1 ; }

# call on a job-id
maxmem_running MY_RUNNING_JOB_ID
maxmem_terminated MY_TERMINATED_JOB_ID