DYAMOND Winter

Simulation Period:

20 January 2020 - 1 March 2020

with 10 day spin-up peripode and 30 days analysis periode

Field Experiment:

align with the EUREC4a field study.

Experiment:

Global atmosphere-only and coupled atmosphere-ocean models with a storm resolving grid spacing of 5 km or less. Detailed information can be found in the DYAMOND Winter protocol (pdf).

Initital data is provided at the ESiWACE webpage.

Completion of the simulation had been done untill summer 2022.

Participating models and description

• ARPEGE-nh
• GEM
• GEOS
• GRIST
• ICON
• IFS
• MPAS
• NICAM
• gSAM
• SCREAM
• GFDL X-SHiELD
• Unified Model (UM)

The DYAMOND coordination team at DKRZ was so kind to standardize most of the data sets to simplify their analysis and ease comparisons. As we are still in the process of standardizing FILE NAMES CAN CHANGE AT ANY TIME WITHOUT PRIOR WARNING. Please have a look at the overview of the processed variables for variable names and data availability.

Grid information

For several models, the grid information is not included in the output. Instead it can be found in a grid.nc file. You can find this grid file in the frequency directory fx/ of the directory structure of the data, e.g. /fastdata/ka1081/DYAMOND/data/winter_data/DYAMOND_WINTER/NOAA/SHiELD-3km/DW-ATM/atmos/fx/gn/grid.nc for the SHiELD data.

To associate data with the grid information, cdo will need a -setgrid,GRIDFILENAME, e.g.

cdo -sellonlatbox,0,20,40,60 -setgrid,GRIDFILENAME INFILE OUTFILE

Getting the data

Since the DYAMOND data sets form hundreds of terabytes of data, only parts are available on disk. To retrive DYAMOND Winter data files from tape, DKRZ provides the Python tool outtake. It is setup to download the files (with read only permissions) to /fastdata/ka1081/DYAMOND/data/winter_data, thereby making it accessible to all DYAMOND users. Examples how to use it in a Python notebook or with the command-line script find_files can be found at Loading data from the catalog and Getting file names from intake with a command-line python script. Unused data files will be deleted on a regular basis.

Working with the data

You are most invited to use our systems for analyzing the data (instead of copying them around). Our post-processing project 1153 has some compute resources that can be used for analysis scripts. Therefore, we recommend to store project data under /work/bb1153/<user-id>/ and use the temporary storage in your scratch folder /scratch/*/<user-id> for processing of large data sets. For more information about the Levantes file systhem its quotas and backups, see The file systems of Levante (docs.dkrz.de).

For model and run descriptions see Participating models and description above.

Using DKRZs JupyterHub and intake-esm

DKRZ provides a JupytherHub server to analyse the big data sets stored at DKRZ by using Python, R or Julia scripts running directly on Levante. For further information visit our page DKRZs JupyterHub.

All post-processed DYAMOND Winter data sets are included in the intake-esm catalog of the DYAMOND and nextGEMS project: /work/ka1081/Catalogs/dyamond-nextgems.json. Intake-esm is a Python library that allows you to easily access data from a variety of simulations in a consistant manner. Take a look our various Pythons scripts!

• Working with Intake and Intake-ESM
  • Tutorials for Intake
    • Listing the entries in an intake catalog
    • Finding files on the command line with query_yaml
    • Searching catalog metadata with search_yaml
  • Tutorials for Intake-ESM
    • Loading data from the catalog
    • Working with Intake-ESM and the Tape archive
    • Map Plotting with Intake and datashader
    • Caching results of computations for later use
    • Accessing the metadata of variables
    • Calculating zonal means
    • Using intake together with CDO
  • Utilities
    • Getting file names from intake with a command-line python script
    • Some helper functions
  • Special cases:
    • Robust Script for getting variable metadata
    • Plotting 2m air temperature for all DYAMOND WINTER simulations

• Create a cdo / paraview compatible grid file from a raw geos file

Direct file access

All stadardized data sets can be found in The DYAMOND Data Library at /fastdata/ka1081/DYAMOND/data/winter_data/ stored at the DKRZ’s Levante supercomputer. The data sets are read-only, so you will have to define the respective subfolder as the source directory for your post-processing scripts.

Browse the data by using our intake-esm catalog

All post-processed DYAMOND Winter data sets are included in the intake-esm catalog of the DYAMOND and nextGEMS project: /work/ka1081/Catalogs/dyamond-nextgems.json.

• Getting file names from intake with a command-line python script

Browse the data by using GEMS

For an overview of the processed files, please have a look at the GEMS webfrontend (use your mistral ID or the “guest” button to log in).

To browse the files in the shell, you can use the freva command line interface:

module load clint gems

# use tab-completion
freva databrowser project=dyamond_winter experiment= TAB TAB

# all surface air temperature files for ifs-4km with 1hr time step
freva databrowser project=dyamond_winter product=dw-cpl model=ifs-4km variable=tas time_frequency=1hr

# call cdo on multiple files
cdo -sinfov $(freva databrowser project=dyamond_winter product=dw-atm model=arpege-nh-2km variable=rst)

See also freva databrowser --help and the comand given at the GEMS webfrontend.

To brows the files in a Python notebook, see the example on the freva documentation webpage.