The DYAMOND Data Library#

The Library structure#

The DYAMOND Data Library is located in /fastdata/ka1081/DYAMOND/, while most of the data is archived in the DKRZ tape system. The library is structured as follows:

  • data/ contains DYAMOND data sets according to the latest data requests

  • scripts_DYAMOND_summer/ contains some example post-processing scripts for DYAMOND Summer data sets

  • indices/ contains the lists of the data archived in DKRZ’s tape archive

  • requests/ – in this folder, requests for data can be made

Since the DYAMOND data sets contain hundreds of terabytes of data, most of the data is archived in DKRZ’s tape archive. To access the data sets associated with the project, you will need to make a request. The ESiWACE team at DKRZ will retrieve the data for you to the data/ folder, thereby making it accessible to all DYAMOND users. Since the data directories are read-only, you will have to keep your post-processed files in separate directories. In order to stay within the quota, unused data sets will be deleted from the data/ folder on a regular basis.

For any questions, please contact us via dyamond@esiwace.eu

Accessing DYAMOND stored at DKRZ tape system#

Available DYAMOND Summer and DYAMOND Winter files on diks can be found in data/. If you miss some files or even a whole data set, please search for the files by using listems and then request the retrieval as described below.

Searching for a data set#

Load listems#

Use the listems tool, which is part of the packems module. Further information about packems and listems can be found in the documentation and in a pad.

  • load the module file

    module add packems

  • check if your StrongLink token is valid

    tapeinit

If you do not have access to the tape library please see the documentation for slk login and known issues on slk.

Search for your files using indices#

Searching files by listems needs indices of the archived files. Archive indices of the DYAMOND Summer collection can be found in /fastdata/ka1081/DYAMOND/indices/summer_data and for the DYAMOND Winter collection in /fastdata/ka1081/DYAMOND/indices/winter_data.

You can search in the full index list /fastdata/ka1081/DYAMOND/indices/summer_data/full_index, which contains all entries of all index lists. e.g: searching for a file named v200_C3072_144x72.fre.nc contained in a tarball named with *FV3-3.25km*

listems -l /fastdata/ka1081/DYAMOND/indices/summer_data/full_index 'v200_C3072_144x72.fre.nc' -a *FV3-3.25km*

To see what else is archived in specific tarballs, e.g: *FV3-3.25km*, type:

listems -l /fastdata/ka1081/DYAMOND/indices/summer_data/full_index -a '*FV3-3.25km*'

Notes:

  • the asterisk allows you to search similar named files

  • for a better readability of the command output, you can append | awk {'print $1" t"$3'} This command lists the searched files and the corresponding tarballs, that they are archived in.

Limiting the search to specific index files#

If you know which model / resolution, you are looking for, you can use specific sub-indices. To search for a file named nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T.grb of the ICON-10km data-set in the DYAMOND Summer collection you can limit your search to the archive index /fastdata/ka1081/DYAMOND/indices/summer_data/ICON-10km by typing (note the -i instead of -l):

listems -i /fastdata/ka1081/DYAMOND/indices/summer_data/ICON-10km 'nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T*'

For further information how to use listems, see the manual pages: listems --help

Request a data set#

  1. Write a file in /fastdata/ka1081/DYAMOND/requests/ folder named: request_<YOUR_ID>

  2. List here the archive path of your desired tarballs, as given by listems.

An example request can be found in /fastdata/ka1081/DYAMOND/requests/example_request_k555555 The content of this folder will be delivered to the ESiWACE project office by e-mail every night. We will then download this data for you to the folder /fastdata/ka1081/DYAMOND/data/, where it will be accessible (read-only) to all DYAMOND users.