The DYAMOND Data Library

The Library Structure

Since the DYAMOND data sets contain several petabytes of data, most of the data is archived in DKRZ’s tape archive. To access the data sets associated with the project, we provide the get_dyamond_summer and get_dyamond_winter tools (see below). Files will be downloaded to the levante hard disks, and shared among all DYAMOND users. Please note that the quota for this storage is limited when downloading large amounts of data. To save space, data sets will be removed from the disks after two weeks without access.

For any questions, please contact us at dyamond@esiwace.eu.

Searching for a Data Set

Load the hsm-tools module

To use get_dyamond_summer and get_dyamond_winter, you first need to load the hsm-tools module. Run the following commands, or add them to your .bash_profile to load the module automatically when you log in.

module use /work/k20200/k202134/hsm-tools/outtake/module
module load hsm-tools/unstable

Use atlas login to authenticate with the tape library:

atlas login

If you do not have access to the tape library, please see the documentation for our hsm.

Search and retrieve your files

get_dyamond_summer and get_dyamond_winter use the same syntax. For brevity, we will only show examples for get_dyamond_summer here. The same commands apply to get_dyamond_winter. Searching for files with get_dyamond_summer is done using regex syntax (.* for matching a sequence of characters). Downloaded files will be printed to stdout for piping and re-using in other commands, any other output goes to stderr. Here are some examples:

# Get the description files of all runs
get_dyamond_summer datadescription.txt 

# get all files containing MPAS-3.75km/history.2016-08-02
get_dyamond_summer MPAS-3.75km/history.2016-08-02

# A more free search using .* for filling a gap in the filename
get_dyamond_summer FV3-3.25km.*v200_C3072_144x72.fre.nc 

# Anything from FV3-3.25km, 2>&1 redirects stderr to stdout for less
get_dyamond_summer FV3-3.25km 2>&1 |less

The above commands will return a list of files that match the search string. To retrieve the files, add the --get flag to the command.

get_dyamond_summer --get MPAS-3.75km/history.2016-08-02
get_dyamond_summer --get FV3-3.25km.*v200_C3072_144x72.fre.nc

The download may take hours to days to complete. Once you have confirmed the size of the download, things will start running in the background, and if you lose your connection, you can just pick up by re-issuing the command a few hours later / the next day.

Warning

Files in the scratch directory where get_dyamond_summer downloads to will be deleted after two weeks without access. Download and process your data promptly.

Note

get_dyamond_summer requires regular expressions to search for files. Please use regex syntax when searching for patterns. For example, instead of only an asterisk (*), use a dot before (.*) to search for any characters containing a specific string.