.. _DYAMOND-winter: DYAMOND Winter ======================================== **Simulation Period:** 20 January 2020 - 1 March 2020 with 10 day spin-up peripode and 30 days analysis periode **Field Experiment:** align with the `EUREC4a field study `_. **Experiment:** Global atmosphere-only and coupled atmosphere-ocean models with a storm resolving grid spacing of 5 km or less. Detailed information can be found in the `DYAMOND Winter protocol (pdf) `_. Initital data is provided at the `ESiWACE webpage `__\. Completion of the simulation had been done untill summer 2022. .. _DYAMOND-winter_participating_models_and_description: Participating models and description ------------------------------------ .. toctree:: :maxdepth: 1 :glob: Models/description/* The DYAMOND coordination team at DKRZ was so kind to standardize most of the data sets to simplify their analysis and ease comparisons. As we are still in the process of standardizing **FILE NAMES CAN CHANGE AT ANY TIME WITHOUT PRIOR WARNING**. Please have a look at the `overview of the processed variables <../../_static/DYAMOND/WINTER/variable_table.html>`_ for variable names and data availability. .. toctree:: :maxdepth: 1 :caption: Data description for standardized model runs: :hidden: Models/ARPEGE-NH-2km.rst Models/GEOS-3km.rst Models/ICON-NWP-2km.rst Models/ICON-SAP-5km.rst Models/ICON-SAP-5km-dpp0029.rst Models/IFS-4km.rst Models/IFS-9km.rst Models/NICAM-3km.rst Models/SAM2-4km.rst Models/SCREAM-3km.rst Models/SHiELD-3km.rst Models/UM-5km.rst .. admonition:: Grid information For several models, the grid information is not included in the output. Instead it can be found in a :file:`grid.nc` file. You can find this grid file in the `frequency` directory :file:`fx/` of the directory structure of the data, e.g. :file:`/fastdata/ka1081/DYAMOND/data/winter_data/DYAMOND_WINTER/NOAA/SHiELD-3km/DW-ATM/atmos/fx/gn/grid.nc` for the SHiELD data. To associate data with the grid information, :code:`cdo` will need a :code:`-setgrid,GRIDFILENAME`, e.g. .. code:: bash cdo -sellonlatbox,0,20,40,60 -setgrid,GRIDFILENAME INFILE OUTFILE Getting the data -------------------------- Since the DYAMOND data sets form hundreds of terabytes of data, only parts are available on disk. **To retrive DYAMOND Winter data files from tape,** DKRZ provides the Python tool *outtake*. It is setup to download the files (with read only permissions) to ``/fastdata/ka1081/DYAMOND/data/winter_data``, thereby making it accessible to all DYAMOND users. Examples how to use it in a Python notebook or with the command-line script `find_files` can be found at :doc:`../../../../Processing/Intake/catalog-basics` and :ref:`intake-find-files-command-line`. Unused data files will be deleted on a regular basis. Working with the data --------------------- You are most invited to use our systems for analyzing the data (instead of copying them around). Our post-processing project 1153 has some compute resources that can be used for analysis scripts. Therefore, we recommend to store project data under ``/work/bb1153//`` and use the temporary storage in your scratch folder ``/scratch/*/`` for processing of large data sets. For more information about the Levantes file systhem its quotas and backups, see `The file systems of Levante (docs.dkrz.de) `__. For model and run descriptions see :ref:`Participating models and description ` above. Using DKRZs JupyterHub and intake-esm ++++++++++++++++++++++++++++++++++++++ DKRZ provides a JupytherHub server to analyse the big data sets stored at DKRZ by using Python, R or Julia scripts running directly on Levante. For further information visit our page :ref:`dkrzs_jupyterhub`. All post-processed DYAMOND Winter data sets are included in the intake-esm catalog of the DYAMOND and nextGEMS project: ``/work/ka1081/Catalogs/dyamond-nextgems.json``. `Intake-esm `__ is a Python library that allows you to easily access data from a variety of simulations in a consistant manner. **Take a look our various Pythons scripts!** .. toctree:: :maxdepth: 3 :glob: /Processing/Intake/index .. toctree:: :maxdepth: 1 :glob: /DYAMOND/Models/*/* /Processing/For-Beginners/index Direct file access ++++++++++++++++++ All stadardized data sets can be found in :ref:`DYAMOND-library` at :file:`/fastdata/ka1081/DYAMOND/data/winter_data/` stored at the DKRZ’s Levante supercomputer. The **data sets are read-only**, so you will have to define the respective subfolder as the source directory for your post-processing scripts. **Browse the data by using our intake-esm catalog** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All post-processed DYAMOND Winter data sets are included in the intake-esm catalog of the DYAMOND and nextGEMS project: ``/work/ka1081/Catalogs/dyamond-nextgems.json``. .. toctree:: :maxdepth: 1 :glob: /Processing/Intake/find_files .. [Julia] The follwoing seems to be outdated!!! the links in /work/ka1081/gems.modules/ are broken. **Browse the data by using GEMS** ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For an overview of the processed files, please have a look at the `GEMS `__ webfrontend (use your mistral ID or the "guest" button to log in). To browse the files in the shell, you can use the freva command line interface:: module load clint gems # use tab-completion freva databrowser project=dyamond_winter experiment= TAB TAB # all surface air temperature files for ifs-4km with 1hr time step freva databrowser project=dyamond_winter product=dw-cpl model=ifs-4km variable=tas time_frequency=1hr # call cdo on multiple files cdo -sinfov $(freva databrowser project=dyamond_winter product=dw-atm model=arpege-nh-2km variable=rst) See also ``freva databrowser --help`` and the comand given at the `GEMS `__ webfrontend. To brows the files in a Python notebook, see the example on the `freva documentation webpage `__\.