The DYAMOND Data Library#

The Library structure#

The DYAMOND Data Library is located in /fastdata/ka1081/DYAMOND/, while most of the data is archived in the DKRZ tape system. The library is structured as follows:

  • data/ contains DYAMOND data sets according to the latest data requests

  • scripts_DYAMOND_summer/ contains some example post-processing scripts for DYAMOND Summer data sets

  • indices/ contains the lists of the data archived in DKRZ’s tape archive

  • requests/ – in this folder, requests for data can be made (deprecated)

Since the DYAMOND data sets contain hundreds of terabytes of data, most of the DYAMOND Summer data is archived in DKRZ’s tape archive. To access the data sets associated with the project, you will need to use the get_dyamond_summer tool. The files then will be downloaded to /scratch/k/k202134/INTAKE_CACHE/ folder, which is accessible to all DYAMOND users. Please keep in mind that the quota for this folder is limited, when downloading large amounts of data.

In order to stay within the quota, unused data sets will be deleted from the /scratch/k/k202134/INTAKE_CACHE/ folder on a regular basis.

For any questions, please contact us via dyamond@esiwace.eu.

Accessing DYAMOND stored at DKRZ tape system#

Available DYAMOND Summer and DYAMOND Winter files on disks can be found in data/. If you miss some files or even a whole data set, please search for the files by using our new get_dyamond_summer tool and then retrieve the files as described below.

Warning

Please not that the tools used for the below instructions are still under development and may change in the future. When faced with any issues, please contact us via dyamond@esiwace.eu.

Searching for a data set#

Load hsm-tools module#

In order to use the get_dyamond_summer tool, you first need to load hsm-tools module. You can run the following commands to load the module and access to the get_dyamond_summer tool or put them into your .bash_profile file to load the module automatically when you log in. Since there may be conflicts between different versions of the slk at the moment, we recommend to unload packems from modules before proceeding:

export hsm_tools='/work/k20200/k202134/hsm-tools'
module use $hsm_tools/outtake/module
module load hsm-tools/unstable

Then check if your StrongLink token is valid:

slk_helpers session

If you do not have access to the tape library please see the documentation for slk login and known issues on slk.

Search and retrieve your files using get_dyamond_summer#

Searching files by get_dyamond_summer is done by using the following command:

get_dyamond_summer MPAS-3.75km/history.2016-08-02
get_dyamond_summer FV3-3.25km.*v200_C3072_144x72.fre.nc
get_dyamond_summer FV3-3.25km.*   # Yields many results

The above commands will return a list of files that match the search string. If you want to retrieve the files, you can add the --get flag to the command as well as sbatch to submit the job to the batch system:

sbatch get_dyamond_summer --get MPAS-3.75km/history.2016-08-02
sbatch get_dyamond_summer --get FV3-3.25km.*v200_C3072_144x72.fre.nc

The download then will run in the background. You can check the status of the running job via squeue -me and by looking at the log file it creates get_dyamond_summer.log$SLURM_JOB_ID. The data will be downloaded to the /scratch/k/k202134/INTAKE_CACHE/ folder. You can run the command again without the --get flag to see the files that are already downloaded.

Note

slk requires regular expressions to search for the files. Therefore please use regex syntax when looking up for patterns. For example instead of only an asterisk (*) you need to use a dot before (.*) to search for any characters containing a specific string.