The DYAMOND Data Library

The Library structure

The DYAMOND Data Library is located in /work/bk1040/DYAMOND/, while most of the data is archived in the DKRZ tape system. The library is structured as follows:

  • data/ contains DYAMOND data sets according to the latest data requests

  • scripts/ contains some example post-processing scripts

  • indices/ contains the lists of the data archived in DKRZ’s tape archive

  • requests/ – in this folder, requests for data can be made

Since the DYAMOND data sets contain hundreds of terabytes of data, most of the data is archived in DKRZ’s tape archive. To access the data sets associated with the project, you will need to make a request. The ESiWACE team at DKRZ will download the data for you to the data/ folder, thereby making it accessible to all DYAMOND users. Since the data directories are read-only, you will have to keep your post-processed files in separate directories. In order not to stay within the quota, unused data sets will be deleted from the data/ folder on a regular basis.

For any questions, please contact us via dyamond@easiwace.eu

Accessing DYAMOND data

First search for the data using listems and then request the download of the files as described below:

Searching for a data set

Load listems

Use the listems tool, which is part of the packems module. Further information about packems and listems can be found in the documentation and in a pad.

  • load the module file

    module add packems
    
  • renew your kerberos ticket

    tapeinit
    

If you do not have access to the tape library without entering a password please follow the recommended procedure described at: https://www.dkrz.de/up/systems/hpss/pftp-with-kerberos.

Search for your files using indices in /work/bk1040/DYAMOND/indices/

Searching files by listems needs indices of the archived files. Archive indices of the DYAMOND Summer collection can be found in /work/bk1040/DYAMOND/indices/summer_data/ and for the DYAMOND Winter collection in /work/bk1040/DYAMOND/indices/winter_data .

You can search in the full index list /work/bk1040/DYAMOND/indices/summer_data/full_index, which contains all entries of all index lists. e.g: searching for a file named v200_C3072_144x72.fre.nc contained in a tarball named with *FV3-3.25km*

listems -l /work/bk1040/DYAMOND/indices/summer_data/full_index 'v200_C3072_144x72.fre.nc' -a *FV3-3.25km*

To see what else is archived in specific tarballs, e.g: *FV3-3.25km*, type:

listems -l /work/bk1040/DYAMOND/indices/summer_data/full_index -a '*FV3-3.25km*'

Notes:

  • the asterisk allows you to search similar named files

  • for a better readability of the comand output, you can append | awk {'print $1" \t"$3'} This comand lists the searched files and the corresponding tarballs, that they are archived in.

Limiting the search to specific index files

If you know which model / resolution, you are looking for, you can use specific sub-indices. To search for a file named nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T.grb of the ICON-10km data-set in the DYAMOND Summer collection you can limit your search to the archive index /work/bk1040/DYAMOND/indices/summer_data/ICON-10km by typing (note the -i instead of -l):

listems -i /work/bk1040/DYAMOND/indices/summer_data/ICON-10km 'nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T*'

For further information how to use listems, see the manual pages: listems --help

To request a data set

  1. Write a file in /work/bk1040/DYAMOND/requests/ folder named: request_<YOUR_ID>

  2. List here the archive path of your desired tarballs, as given by listems.

An example request can be found in /work/bk1040/DYAMOND/requests/example_request_k555555 The content of this folder will be delivered to the ESiWACE project office by e-mail every night. We will then download this data for you to the folder /work/bk1040/DYAMOND/data/, where it will be accessible (read-only) to all DYAMOND users.