.. _DYAMOND-library: The DYAMOND Data Library ======================== The Library structure --------------------- The DYAMOND Data Library is located in :file:`/fastdata/ka1081/DYAMOND/`, while most of the data is archived in the DKRZ tape system. The library is structured as follows: * :file:`data/` contains DYAMOND data sets according to the latest data requests * :file:`scripts_DYAMOND_summer/` contains some example post-processing scripts for **DYAMOND Summer** data sets * :file:`indices/` contains the lists of the data archived in DKRZ's tape archive * :file:`requests/` -- in this folder, requests for data can be made Since the DYAMOND data sets contain hundreds of terabytes of data, most of the data is archived in DKRZ's tape archive. To access the data sets associated with the project, you will need to make a request. The ESiWACE team at DKRZ will retrieve the data for you to the :file:`data/` folder, thereby making it accessible to all DYAMOND users. Since the data directories are read-only, you will have to keep your post-processed files in separate directories. In order to stay within the quota, unused data sets will be deleted from the :file:`data/` folder on a regular basis. For any questions, please contact us via dyamond@esiwace.eu Accessing DYAMOND stored at DKRZ tape system ---------------------------------------------- Available **DYAMOND Summer** and **DYAMOND Winter** files on diks can be found in :file:`data/`. If you miss some files or even a whole data set, please search for the files by using listems and then request the retrieval as described below. Searching for a data set ++++++++++++++++++++++++ **Load listems** ~~~~~~~~~~~~~~~~ Use the listems tool, which is part of the packems module. Further information about packems and listems can be found in the `documentation `_ and in a `pad `_. * load the module file :: module add packems * check if your StrongLink token is valid :: tapeinit If you do not have access to the tape library please see the documentation for `slk login `_ and `known issues on slk `_. **Search for your files using indices** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Searching files by listems needs indices of the archived files. Archive indices of the **DYAMOND Summer** collection can be found in :file:`/fastdata/ka1081/DYAMOND/indices/summer_data` and for the **DYAMOND Winter** collection in :file:`/fastdata/ka1081/DYAMOND/indices/winter_data`. You can search in the full index list :file:`/fastdata/ka1081/DYAMOND/indices/summer_data/full_index`, which contains all entries of all index lists. e.g: searching for a file named :file:`v200_C3072_144x72.fre.nc` contained in a tarball named with :file:`*FV3-3.25km*` .. code:: bash listems -l /fastdata/ka1081/DYAMOND/indices/summer_data/full_index 'v200_C3072_144x72.fre.nc' -a *FV3-3.25km* To see what else is archived in specific tarballs, e.g: :file:`*FV3-3.25km*`, type: .. code:: bash listems -l /fastdata/ka1081/DYAMOND/indices/summer_data/full_index -a '*FV3-3.25km*' Notes: * the asterisk allows you to search similar named files * for a better readability of the command output, you can append :code:`| awk {'print $1" \t"$3'}` This command lists the searched files and the corresponding tarballs, that they are archived in. **Limiting the search to specific index files** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you know which model / resolution, you are looking for, you can use specific sub-indices. To search for a file named :file:`nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T.grb` of the ICON-10km data-set in the **DYAMOND Summer** collection you can limit your search to the archive index :file:`/fastdata/ka1081/DYAMOND/indices/summer_data/ICON-10km` by typing (note the :code:`-i` instead of :code:`-l`): .. code:: bash listems -i /fastdata/ka1081/DYAMOND/indices/summer_data/ICON-10km 'nwp_R2B08_lkm1012_atm_3d_w_ml_20160909T*' For further information how to use listems, see the manual pages: :code:`listems --help` Request a data set +++++++++++++++++++++ #. Write a file in :file:`/fastdata/ka1081/DYAMOND/requests/` folder named: :file:`request_` #. List here the archive path of your desired tarballs, as given by listems. An example request can be found in :file:`/fastdata/ka1081/DYAMOND/requests/example_request_k555555` The content of this folder will be delivered to the ESiWACE project office by e-mail every night. We will then download this data for you to the folder :file:`/fastdata/ka1081/DYAMOND/data/`, where it will be accessible (read-only) to all DYAMOND users.