Dask chunks and best practices

This notebook is meant to be used in DKRZ’s Jupyterhub with the python3 unstable kernel.

We aim to understand

• the different types of chunks, especially

  • storage chunks

    • The smallest unit of binary data that is stored within a climate data file (zarr, netcdf, grib)

  • dask chunks

    • The chunks that dask uses inside a dask array. The smallest unit for which dask executes a workflow at once.

• the best practices recommended by dask.

We use the following packages which needs to be in your environment/kernel:

import intake
import xarray as xr
import hvplot.xarray
import numpy as np
from xhistogram.xarray import histogram
import dask
from dask.distributed import performance_report
from distributed import Client
import time
import glob

We mainly look at high frequency (30minutes) high resolution (zoom level 10= ICON data from NextGEMS Cycle 3 and select precipitation as an example variable.

We get the data from an intake catalog:

CATALOG_LINK = "https://data.nextgems-h2020.eu/catalog.yaml"
VARID = "pr"

selection = dict(zoom=10, time="PT30M")

cat = intake.open_catalog(CATALOG_LINK)
cat

data_nextgems-h2020_eu:
  args:
    path: https://data.nextgems-h2020.eu/catalog.yaml
  description: ''
  driver: intake.catalog.local.YAMLFileCatalog
  metadata: {}

Chunks

TLDR,

Dask chunks need to

• be smaller than your memory

Dask chunks should

• be about 100MB

• not exceed about O(1 mio) in total when sent to the scheduler i.e. when computed

• reflect a multiple of the storage chunks

• not cut storage chunks

• fit to your analysis

with open_zarr and intake configurations

In the following, we have a look on the NGC3 zarr data and open it in 3 different ways:

1. The catalog is configured such that xarray opens data lazily and encoded as numpy arrays (no dask)

ds_delayed = cat["ICON"]["ngc3028"](**selection).to_dask()[VARID]
ds_delayed

<xarray.DataArray 'pr' (time: 96480, cell: 12582912)>
[1213999349760 values with dtype=float32]
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 12582912

• ...

  [1213999349760 values with dtype=float32]

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

The same as:

ds_delayed = xr.open_dataset(
    f"/fastdata/ka1081/nextgems_cycle3/ngc3028/ngc3028_{selection['time']}_{str(selection['zoom'])}.zarr",
    engine="zarr",
    chunks=None,
)[VARID]

2. Let dask create dask chunks when opening with chunks="auto":

ds_auto = cat["ICON"]["ngc3028"](**selection, chunks="auto").to_dask()[VARID]
ds_auto

<xarray.DataArray 'pr' (time: 96480, cell: 12582912)>
dask.array<open_dataset-dd9675028d7e43a50230dc83a922860apr, shape=(96480, 12582912), dtype=float32, chunksize=(32, 524288), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 12582912

• dask.array<chunksize=(32, 524288), meta=np.ndarray>

  Array Chunk Bytes 4.42 TiB 64.00 MiB Shape (96480, 12582912) (32, 524288) Dask graph 72360 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

The same as:

ds_auto = xr.open_dataset(
    f"/fastdata/ka1081/nextgems_cycle3/ngc3028/ngc3028_{selection['time']}_{str(selection['zoom'])}.zarr",
    engine="zarr",
    chunks="auto",
)[VARID]

But why is the .encoding["chunks"] different to the dask chunks?

ds_auto.encoding["chunks"]

(16, 262144)

Because the latter are storage chunks. It is the shape of a chunk written to the storage.

Storage chunks have a different optimal size than dask chunks. Each storage chunk can be compressed individually. Therefore, it depends a lot on the chosen compression tool what the optimal size of a storage chunk is. Also, optimal access to the storage backend hardware affects the optimal storage chunk size.

In general, one can say that 1-10MB is a good storage chunk size for disk/lustre and <100MB is still OK for most backends.

3. Force dask to use storage chunks when opening with chunks={}:

ds_storage = cat["ICON"]["ngc3028"](**selection, chunks={}).to_dask()[VARID]
ds_storage

<xarray.DataArray 'pr' (time: 96480, cell: 12582912)>
dask.array<open_dataset-11ac8388882b0270658ade8f13719e04pr, shape=(96480, 12582912), dtype=float32, chunksize=(16, 262144), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 12582912

• dask.array<chunksize=(16, 262144), meta=np.ndarray>

  Array Chunk Bytes 4.42 TiB 16.00 MiB Shape (96480, 12582912) (16, 262144) Dask graph 289440 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

The same as:

ds_storage = xr.open_dataset(
    f"/fastdata/ka1081/nextgems_cycle3/ngc3028/ngc3028_{selection['time']}_{str(selection['zoom'])}.zarr",
    engine="zarr",
    chunks={},
)[VARID]

We look at a

• 5TB array

• which is chunked in all dimensions both in storage chunks and dask chunks

• and results in a dask graph with O(100,000) chunks

Note

To load one individual dask chunk is one task in the dask’s task graph.

When we compare chunks of ds_auto and ds_storage, we find that

• Storage chunks of precipitation are small.

• Dask aims to create chunks with about 100MB in size

• Dask is clever to use a multiple of the storage chunk size:

print(ds_auto.chunksizes["time"][0] / ds_storage.chunksizes["time"][0])
print(ds_auto.chunksizes["cell"][0] / ds_storage.chunksizes["cell"][0])

2.0
2.0

That means, we have 2²=4 storage chunks within one dask chunk.

Dask is able to load these storage chunks at once.

Note

NOTE: chunks("auto") returns well-configured dask chunks for general analysis purpose so that a chunk has about 100MB.

Config and default setting

Sometimes 100MB memory is not the best option for a general approach. You can change this dask setting with:

# dask.config.set({"array.slicing.split_large_chunks": False})
# dask.config.set({"array.chunk-size": "10 MB"})

# find out the target dask chunk size:
dask.config.config["array"]["chunk-size"]

'128MiB'

# set the target dask chunk size:
dask.config.set({"array.chunk-size": "10 MB"})

<dask.config.set at 0x7fff7ccdb2e0>

If we now open the data with chunks="auto", we get:

ds_auto_config = xr.open_dataset(
    f"/fastdata/ka1081/nextgems_cycle3/ngc3028/ngc3028_{selection['time']}_{str(selection['zoom'])}.zarr",
    engine="zarr",
    chunks="auto",
)[VARID]
ds_auto_config

/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/dataset.py:255: UserWarning: The specified Dask chunks separate the stored chunks along dimension "time" starting at index 12. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(
/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/dataset.py:255: UserWarning: The specified Dask chunks separate the stored chunks along dimension "cell" starting at index 208332. This could degrade performance. Instead, consider rechunking after loading.
  warnings.warn(

<xarray.DataArray 'pr' (time: 96480, cell: 12582912)>
dask.array<open_dataset-1a14ada58f9017b077d18ef7caddcac1pr, shape=(96480, 12582912), dtype=float32, chunksize=(12, 208332), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 12582912

• dask.array<chunksize=(12, 208332), meta=np.ndarray>

  Array Chunk Bytes 4.42 TiB 9.54 MiB Shape (96480, 12582912) (12, 208332) Dask graph 490440 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

# set the target dask chunk size:
dask.config.set({"array.chunk-size": "128 MB"})

<dask.config.set at 0x7fff7c69cd00>

open_mfdataset on netcdf

It is getting even more complicated. open_mfdataset behaves differently!

Note

Using chunks=auto does not always lead to dask chunks being a multiple of storage chunks

Consider this example:

varname = "ta"
filelist = glob.glob(
    f"/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp126/r1i1p1f1/CFday/{varname}/gn/v20190710/*.nc"
)

openmf_auto = xr.open_mfdataset(filelist, chunks="auto")
openmf_auto

<xarray.Dataset>
Dimensions:    (time: 31411, bnds: 2, lev: 95, lat: 192, lon: 384)
Coordinates:
  * time       (time) datetime64[ns] 2015-01-01T12:00:00 ... 2100-12-31T12:00:00
  * lev        (lev) float64 0.9961 0.9826 0.959 ... 2.31e-05 9.816e-06
  * lat        (lat) float64 -89.28 -88.36 -87.42 -86.49 ... 87.42 88.36 89.28
  * lon        (lon) float64 0.0 0.9375 1.875 2.812 ... 356.2 357.2 358.1 359.1
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(1826, 2), meta=np.ndarray>
    lev_bnds   (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    ap         (time, lev) float64 dask.array<chunksize=(1826, 95), meta=np.ndarray>
    b          (time, lev) float64 dask.array<chunksize=(1826, 95), meta=np.ndarray>
    ps         (time, lat, lon) float32 dask.array<chunksize=(1142, 118, 237), meta=np.ndarray>
    ap_bnds    (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    b_bnds     (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>
    ta         (time, lev, lat, lon) float32 dask.array<chunksize=(423, 21, 42, 85), meta=np.ndarray>
Attributes: (12/47)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            ScenarioMIP
    branch_method:          standard
    branch_time_in_child:   60265.0
    branch_time_in_parent:  60265.0
    contact:                cmip6-mpi-esm@dkrz.de
    ...                     ...
    title:                  MPI-ESM1-2-HR output prepared for CMIP6
    variable_id:            ta
    variant_label:          r1i1p1f1
    license:                CMIP6 model data produced by DKRZ is licensed und...
    cmor_version:           3.4.0
    tracking_id:            hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

xarray.Dataset

• Dimensions:
  • time: 31411
  • bnds: 2
  • lev: 95
  • lat: 192
  • lon: 384
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2015-01-01T12:00:00 ... 2100-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',
           '2015-01-03T12:00:00.000000000', ..., '2100-12-29T12:00:00.000000000',
           '2100-12-30T12:00:00.000000000', '2100-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9961 0.9826 ... 9.816e-06

    bounds :

    lev_bnds

    units :

    1

    axis :

    Z

    positive :

    down

    long_name :

    hybrid sigma pressure coordinate

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula :

    p = ap + b*ps

    formula_terms :

    ap: ap b: b ps: ps

    array([9.961500e-01, 9.826500e-01, 9.589556e-01, 9.276460e-01, 8.908394e-01,
           8.501503e-01, 8.068475e-01, 7.620137e-01, 7.164562e-01, 6.707188e-01,
           6.252446e-01, 5.803910e-01, 5.366074e-01, 4.942692e-01, 4.537201e-01,
           4.154341e-01, 3.797069e-01, 3.466272e-01, 3.161519e-01, 2.881811e-01,
           2.625506e-01, 2.391084e-01, 2.177225e-01, 1.982337e-01, 1.804680e-01,
           1.642867e-01, 1.495502e-01, 1.361203e-01, 1.238958e-01, 1.127819e-01,
           1.026525e-01, 9.340927e-02, 8.499143e-02, 7.732509e-02, 7.033860e-02,
           6.396105e-02, 5.814545e-02, 5.284837e-02, 4.802120e-02, 4.362085e-02,
           3.960960e-02, 3.595304e-02, 3.261980e-02, 2.958129e-02, 2.681146e-02,
           2.428654e-02, 2.198489e-02, 1.988675e-02, 1.797414e-02, 1.623065e-02,
           1.464207e-02, 1.319605e-02, 1.188111e-02, 1.068652e-02, 9.602347e-03,
           8.619337e-03, 7.728926e-03, 6.923179e-03, 6.194759e-03, 5.536888e-03,
           4.943310e-03, 4.408264e-03, 3.926449e-03, 3.492988e-03, 3.103407e-03,
           2.753606e-03, 2.439827e-03, 2.158631e-03, 1.906879e-03, 1.681707e-03,
           1.480501e-03, 1.300885e-03, 1.140697e-03, 9.979719e-04, 8.709289e-04,
           7.579535e-04, 6.575843e-04, 5.685002e-04, 4.895081e-04, 4.195318e-04,
           3.577273e-04, 3.034278e-04, 2.559719e-04, 2.147130e-04, 1.790279e-04,
           1.483228e-04, 1.220382e-04, 9.965253e-05, 8.068427e-05, 6.469293e-05,
           5.127906e-05, 4.008352e-05, 3.078596e-05, 2.310285e-05, 9.815852e-06])

  • lat
    (lat)
    float64
    -89.28 -88.36 ... 88.36 89.28

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-89.284228, -88.357004, -87.424304, -86.490367, -85.55596 , -84.621327,
           -83.686567, -82.751728, -81.816839, -80.881913, -79.946962, -79.011992,
           -78.077007, -77.142011, -76.207005, -75.271992, -74.336973, -73.401949,
           -72.466921, -71.531889, -70.596854, -69.661816, -68.726776, -67.791734,
           -66.856691, -65.921645, -64.986599, -64.051551, -63.116502, -62.181452,
           -61.246401, -60.311349, -59.376297, -58.441244, -57.50619 , -56.571136,
           -55.636081, -54.701026, -53.76597 , -52.830914, -51.895857, -50.960801,
           -50.025743, -49.090686, -48.155628, -47.22057 , -46.285512, -45.350454,
           -44.415395, -43.480336, -42.545277, -41.610218, -40.675159, -39.740099,
           -38.805039, -37.86998 , -36.93492 , -35.99986 , -35.064799, -34.129739,
           -33.194679, -32.259618, -31.324558, -30.389497, -29.454436, -28.519375,
           -27.584315, -26.649254, -25.714193, -24.779132, -23.84407 , -22.909009,
           -21.973948, -21.038887, -20.103825, -19.168764, -18.233703, -17.298641,
           -16.36358 , -15.428518, -14.493457, -13.558395, -12.623333, -11.688272,
           -10.75321 ,  -9.818148,  -8.883087,  -7.948025,  -7.012963,  -6.077901,
            -5.14284 ,  -4.207778,  -3.272716,  -2.337654,  -1.402593,  -0.467531,
             0.467531,   1.402593,   2.337654,   3.272716,   4.207778,   5.14284 ,
             6.077901,   7.012963,   7.948025,   8.883087,   9.818148,  10.75321 ,
            11.688272,  12.623333,  13.558395,  14.493457,  15.428518,  16.36358 ,
            17.298641,  18.233703,  19.168764,  20.103825,  21.038887,  21.973948,
            22.909009,  23.84407 ,  24.779132,  25.714193,  26.649254,  27.584315,
            28.519375,  29.454436,  30.389497,  31.324558,  32.259618,  33.194679,
            34.129739,  35.064799,  35.99986 ,  36.93492 ,  37.86998 ,  38.805039,
            39.740099,  40.675159,  41.610218,  42.545277,  43.480336,  44.415395,
            45.350454,  46.285512,  47.22057 ,  48.155628,  49.090686,  50.025743,
            50.960801,  51.895857,  52.830914,  53.76597 ,  54.701026,  55.636081,
            56.571136,  57.50619 ,  58.441244,  59.376297,  60.311349,  61.246401,
            62.181452,  63.116502,  64.051551,  64.986599,  65.921645,  66.856691,
            67.791734,  68.726776,  69.661816,  70.596854,  71.531889,  72.466921,
            73.401949,  74.336973,  75.271992,  76.207005,  77.142011,  78.077007,
            79.011992,  79.946962,  80.881913,  81.816839,  82.751728,  83.686567,
            84.621327,  85.55596 ,  86.490367,  87.424304,  88.357004,  89.284228])

  • lon
    (lon)
    float64
    0.0 0.9375 1.875 ... 358.1 359.1

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.    ,   0.9375,   1.875 , ..., 357.1875, 358.125 , 359.0625])

• Data variables: (10)
  • time_bnds
    (time, bnds)
    datetime64[ns]
    dask.array<chunksize=(1826, 2), meta=np.ndarray>

    Array Chunk Bytes 490.80 kiB 28.55 kiB Shape (31411, 2) (1827, 2) Dask graph 18 chunks in 37 graph layers Data type datetime64[ns] numpy.ndarray

  • lev_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    formula :

    p = ap + b*ps

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    units :

    1

    formula_terms :

    ap: ap_bnds b: b_bnds ps: ps

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ap
    (time, lev)
    float64
    dask.array<chunksize=(1826, 95), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k)

    units :

    Pa

    Array Chunk Bytes 22.77 MiB 1.32 MiB Shape (31411, 95) (1827, 95) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b
    (time, lev)
    float64
    dask.array<chunksize=(1826, 95), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k)

    Array Chunk Bytes 22.77 MiB 1.32 MiB Shape (31411, 95) (1827, 95) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ps
    (time, lat, lon)
    float32
    dask.array<chunksize=(1142, 118, 237), meta=np.ndarray>

    long_name :

    Surface Air Pressure

    units :

    Pa

    Array Chunk Bytes 8.63 GiB 121.83 MiB Shape (31411, 192, 384) (1142, 118, 237) Dask graph 140 chunks in 38 graph layers Data type float32 numpy.ndarray

  • ap_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k+1/2)

    units :

    Pa

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k+1/2)

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>

    Array Chunk Bytes 92.02 MiB 5.35 MiB Shape (31411, 192, 2) (1827, 192, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>

    Array Chunk Bytes 184.05 MiB 10.71 MiB Shape (31411, 384, 2) (1827, 384, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ta
    (time, lev, lat, lon)
    float32
    dask.array<chunksize=(423, 21, 42, 85), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Air Temperature

    comment :

    Air Temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-20T10:06:39Z altered by CMOR: Reordered dimensions, original order: time lat lon lev. 2019-07-20T10:06:39Z altered by CMOR: replaced missing value flag (-9e+33) with standard missing value (1e+20). 2019-07-20T10:06:40Z altered by CMOR: Converted type from 'd' to 'f'. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lev. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lat.

    Array Chunk Bytes 819.59 GiB 120.97 MiB Shape (31411, 95, 192, 384) (423, 21, 42, 85) Dask graph 34496 chunks in 55 graph layers Data type float32 numpy.ndarray

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-01-01 12:00:00', '2015-01-02 12:00:00',
                   '2015-01-03 12:00:00', '2015-01-04 12:00:00',
                   '2015-01-05 12:00:00', '2015-01-06 12:00:00',
                   '2015-01-07 12:00:00', '2015-01-08 12:00:00',
                   '2015-01-09 12:00:00', '2015-01-10 12:00:00',
                   ...
                   '2100-12-22 12:00:00', '2100-12-23 12:00:00',
                   '2100-12-24 12:00:00', '2100-12-25 12:00:00',
                   '2100-12-26 12:00:00', '2100-12-27 12:00:00',
                   '2100-12-28 12:00:00', '2100-12-29 12:00:00',
                   '2100-12-30 12:00:00', '2100-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='time', length=31411, freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([    0.9961499869823456,     0.9826499819755554,     0.9589555908567136,
               0.9276459693502798,     0.8908393983675643,     0.8501502852051046,
               0.8068474689019394,     0.7620136942249918,     0.7164562136862798,
               0.6707187556893979,     0.6252446440590949,     0.5803910498630256,
               0.5366073683506467,     0.4942692315672581,     0.4537200746596845,
               0.4154341220199812,     0.3797068559311656,     0.3466272163038162,
              0.31615186284801977,     0.2881811466818235,     0.2625505749313972,
               0.2391083761673929,     0.2177224959069833,    0.19823367227649222,
              0.18046804676613204,     0.1642866912611011,     0.1495502438573978,
               0.1361203084601691,    0.12389576315665891,     0.1127819063778702,
              0.10265254219114787,    0.09340926514608601,    0.08499143437525487,
              0.07732509098198866,    0.07033859949420182,    0.06396105122748581,
              0.05814545167160128,   0.052848372347643724,    0.04802119957438934,
             0.043620851838144584,    0.03960959667221811,   0.035953036331112756,
              0.03261979576278374,    0.02958129009375771,   0.026811458381522945,
             0.024286543147051566,    0.02198488918316679,    0.01988675441031335,
              0.01797414241765359,   0.016230649461047374,   0.014642066833209968,
             0.013196053174346163,   0.011881106828660868,   0.010686522473707747,
             0.009602346544997533,   0.008619337479567604,   0.007728925959551567,
             0.006923178769969776,   0.006194759042298915,    0.00553688770278189,
             0.004943309739698989,    0.00440826408485998,   0.003926448676134962,
            0.0034929879294658276,  0.0031034074393196396,   0.002753606269661362,
             0.002439826760149696,   0.002158630883108384,  0.0019068793864594898,
            0.0016817067200655764,  0.0014805012328107652,  0.0013008852943855245,
            0.0011406967724043146,  0.0009979718648560865,  0.0008709289114062885,
            0.0007579534847354583,  0.0006575843101367814,  0.0005685002392614267,
            0.0004895081274191186, 0.00041953176502648504,  0.0003577273067885671,
           0.00030342777834802155, 0.00025597185256773534, 0.00021471303567573402,
            0.0001790279498548529, 0.00014832281342795124, 0.00012203819864095655,
            9.965253293940734e-05,  8.068427314824185e-05,  6.469292667745632e-05,
            5.127906093413877e-05, 4.0083522123670446e-05,  3.078596226644904e-05,
           2.3102853258774366e-05,  9.815852317932415e-06],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([-89.28422753251364, -88.35700351866494, -87.42430374606988,
           -86.49036676628116, -85.55596048489265,  -84.6213271076488,
           -83.68656681656385, -82.75172847343066, -81.81683872860319,
            -80.8819133467975,
           ...
             80.8819133467975,  81.81683872860319,  82.75172847343066,
            83.68656681656385,   84.6213271076488,  85.55596048489265,
            86.49036676628116,  87.42430374606988,  88.35700351866494,
            89.28422753251364],
          dtype='float64', name='lat', length=192))

  • lon
    PandasIndex

    PandasIndex(Index([     0.0,   0.9375,    1.875,   2.8125,     3.75,   4.6875,    5.625,
             6.5625,      7.5,   8.4375,
           ...
            350.625, 351.5625,    352.5, 353.4375,  354.375, 355.3125,   356.25,
           357.1875,  358.125, 359.0625],
          dtype='float64', name='lon', length=384))

• Attributes: (47)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  ScenarioMIP

  branch_method :

  standard

  branch_time_in_child :

  60265.0

  branch_time_in_parent :

  60265.0

  contact :

  cmip6-mpi-esm@dkrz.de

  creation_date :

  2019-07-20T10:06:40Z

  data_specs_version :

  01.00.30

  experiment :

  update of RCP2.6 based on SSP1

  experiment_id :

  ssp126

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  day

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.DKRZ.MPI-ESM1-2-HR.ssp126.none.r1i1p1f1

  grid :

  gn

  grid_label :

  gn

  history :

  2019-07-20T10:06:40Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.

  initialization_index :

  1

  institution :

  Deutsches Klimarechenzentrum, Hamburg 20146, Germany

  institution_id :

  DKRZ

  mip_era :

  CMIP6

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  historical

  parent_mip_era :

  CMIP6

  parent_source_id :

  MPI-ESM1-2-HR

  parent_time_units :

  days since 1850-1-1 00:00:00

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  project_id :

  CMIP6

  realization_index :

  1

  realm :

  atmos

  references :

  MPI-ESM: Mauritsen, T. et al. (2019), Developments in the MPI‐M Earth System Model version 1.2 (MPI‐ESM1.2) and Its Response to Increasing CO2, J. Adv. Model. Earth Syst.,11, 998-1038, doi:10.1029/2018MS001400, Mueller, W.A. et al. (2018): A high‐resolution version of the Max Planck Institute Earth System Model MPI‐ESM1.2‐HR. J. Adv. Model. EarthSyst.,10,1383–1413, doi:10.1029/2017MS001217

  source :

  MPI-ESM1.2-HR (2017): aerosol: none, prescribed MACv2-SP atmos: ECHAM6.3 (spectral T127; 384 x 192 longitude/latitude; 95 levels; top level 0.01 hPa) atmosChem: none land: JSBACH3.20 landIce: none/prescribed ocean: MPIOM1.63 (tripolar TP04, approximately 0.4deg; 802 x 404 longitude/latitude; 40 levels; top grid cell 0-12 m) ocnBgchem: HAMOCC6 seaIce: unnamed (thermodynamic (Semtner zero-layer) dynamic (Hibler 79) sea ice model)

  source_id :

  MPI-ESM1-2-HR

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  CFday

  table_info :

  Creation Date:(09 May 2019) MD5:e6ef8ececc8f338646ebfb3aeed36bfc

  title :

  MPI-ESM1-2-HR output prepared for CMIP6

  variable_id :

  ta

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by DKRZ is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

This configuration gives us chunks which split the storage chunks.

storage_chunks = openmf_auto[varname].encoding["chunksizes"]
storage_chunks

(1, 48, 96, 192)

This is the second worst configuration you can have in dask. On top of that, chunks={} does not gives us storage chunks either but one file per chunk:

one_file_per_chunk = xr.open_mfdataset(
    "/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp126/r1i1p1f1/CFday/ta/gn/v20190710/*.nc",
    chunks={},
)
one_file_per_chunk

<xarray.Dataset>
Dimensions:    (time: 31411, bnds: 2, lev: 95, lat: 192, lon: 384)
Coordinates:
  * time       (time) datetime64[ns] 2015-01-01T12:00:00 ... 2100-12-31T12:00:00
  * lev        (lev) float64 0.9961 0.9826 0.959 ... 2.31e-05 9.816e-06
  * lat        (lat) float64 -89.28 -88.36 -87.42 -86.49 ... 87.42 88.36 89.28
  * lon        (lon) float64 0.0 0.9375 1.875 2.812 ... 356.2 357.2 358.1 359.1
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(1826, 2), meta=np.ndarray>
    lev_bnds   (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    ap         (time, lev) float64 dask.array<chunksize=(1826, 95), meta=np.ndarray>
    b          (time, lev) float64 dask.array<chunksize=(1826, 95), meta=np.ndarray>
    ps         (time, lat, lon) float32 dask.array<chunksize=(1826, 192, 384), meta=np.ndarray>
    ap_bnds    (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    b_bnds     (time, lev, bnds) float64 dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>
    ta         (time, lev, lat, lon) float32 dask.array<chunksize=(1826, 95, 192, 384), meta=np.ndarray>
Attributes: (12/47)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            ScenarioMIP
    branch_method:          standard
    branch_time_in_child:   60265.0
    branch_time_in_parent:  60265.0
    contact:                cmip6-mpi-esm@dkrz.de
    ...                     ...
    title:                  MPI-ESM1-2-HR output prepared for CMIP6
    variable_id:            ta
    variant_label:          r1i1p1f1
    license:                CMIP6 model data produced by DKRZ is licensed und...
    cmor_version:           3.4.0
    tracking_id:            hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

xarray.Dataset

• Dimensions:
  • time: 31411
  • bnds: 2
  • lev: 95
  • lat: 192
  • lon: 384
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2015-01-01T12:00:00 ... 2100-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',
           '2015-01-03T12:00:00.000000000', ..., '2100-12-29T12:00:00.000000000',
           '2100-12-30T12:00:00.000000000', '2100-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9961 0.9826 ... 9.816e-06

    bounds :

    lev_bnds

    units :

    1

    axis :

    Z

    positive :

    down

    long_name :

    hybrid sigma pressure coordinate

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula :

    p = ap + b*ps

    formula_terms :

    ap: ap b: b ps: ps

    array([9.961500e-01, 9.826500e-01, 9.589556e-01, 9.276460e-01, 8.908394e-01,
           8.501503e-01, 8.068475e-01, 7.620137e-01, 7.164562e-01, 6.707188e-01,
           6.252446e-01, 5.803910e-01, 5.366074e-01, 4.942692e-01, 4.537201e-01,
           4.154341e-01, 3.797069e-01, 3.466272e-01, 3.161519e-01, 2.881811e-01,
           2.625506e-01, 2.391084e-01, 2.177225e-01, 1.982337e-01, 1.804680e-01,
           1.642867e-01, 1.495502e-01, 1.361203e-01, 1.238958e-01, 1.127819e-01,
           1.026525e-01, 9.340927e-02, 8.499143e-02, 7.732509e-02, 7.033860e-02,
           6.396105e-02, 5.814545e-02, 5.284837e-02, 4.802120e-02, 4.362085e-02,
           3.960960e-02, 3.595304e-02, 3.261980e-02, 2.958129e-02, 2.681146e-02,
           2.428654e-02, 2.198489e-02, 1.988675e-02, 1.797414e-02, 1.623065e-02,
           1.464207e-02, 1.319605e-02, 1.188111e-02, 1.068652e-02, 9.602347e-03,
           8.619337e-03, 7.728926e-03, 6.923179e-03, 6.194759e-03, 5.536888e-03,
           4.943310e-03, 4.408264e-03, 3.926449e-03, 3.492988e-03, 3.103407e-03,
           2.753606e-03, 2.439827e-03, 2.158631e-03, 1.906879e-03, 1.681707e-03,
           1.480501e-03, 1.300885e-03, 1.140697e-03, 9.979719e-04, 8.709289e-04,
           7.579535e-04, 6.575843e-04, 5.685002e-04, 4.895081e-04, 4.195318e-04,
           3.577273e-04, 3.034278e-04, 2.559719e-04, 2.147130e-04, 1.790279e-04,
           1.483228e-04, 1.220382e-04, 9.965253e-05, 8.068427e-05, 6.469293e-05,
           5.127906e-05, 4.008352e-05, 3.078596e-05, 2.310285e-05, 9.815852e-06])

  • lat
    (lat)
    float64
    -89.28 -88.36 ... 88.36 89.28

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-89.284228, -88.357004, -87.424304, -86.490367, -85.55596 , -84.621327,
           -83.686567, -82.751728, -81.816839, -80.881913, -79.946962, -79.011992,
           -78.077007, -77.142011, -76.207005, -75.271992, -74.336973, -73.401949,
           -72.466921, -71.531889, -70.596854, -69.661816, -68.726776, -67.791734,
           -66.856691, -65.921645, -64.986599, -64.051551, -63.116502, -62.181452,
           -61.246401, -60.311349, -59.376297, -58.441244, -57.50619 , -56.571136,
           -55.636081, -54.701026, -53.76597 , -52.830914, -51.895857, -50.960801,
           -50.025743, -49.090686, -48.155628, -47.22057 , -46.285512, -45.350454,
           -44.415395, -43.480336, -42.545277, -41.610218, -40.675159, -39.740099,
           -38.805039, -37.86998 , -36.93492 , -35.99986 , -35.064799, -34.129739,
           -33.194679, -32.259618, -31.324558, -30.389497, -29.454436, -28.519375,
           -27.584315, -26.649254, -25.714193, -24.779132, -23.84407 , -22.909009,
           -21.973948, -21.038887, -20.103825, -19.168764, -18.233703, -17.298641,
           -16.36358 , -15.428518, -14.493457, -13.558395, -12.623333, -11.688272,
           -10.75321 ,  -9.818148,  -8.883087,  -7.948025,  -7.012963,  -6.077901,
            -5.14284 ,  -4.207778,  -3.272716,  -2.337654,  -1.402593,  -0.467531,
             0.467531,   1.402593,   2.337654,   3.272716,   4.207778,   5.14284 ,
             6.077901,   7.012963,   7.948025,   8.883087,   9.818148,  10.75321 ,
            11.688272,  12.623333,  13.558395,  14.493457,  15.428518,  16.36358 ,
            17.298641,  18.233703,  19.168764,  20.103825,  21.038887,  21.973948,
            22.909009,  23.84407 ,  24.779132,  25.714193,  26.649254,  27.584315,
            28.519375,  29.454436,  30.389497,  31.324558,  32.259618,  33.194679,
            34.129739,  35.064799,  35.99986 ,  36.93492 ,  37.86998 ,  38.805039,
            39.740099,  40.675159,  41.610218,  42.545277,  43.480336,  44.415395,
            45.350454,  46.285512,  47.22057 ,  48.155628,  49.090686,  50.025743,
            50.960801,  51.895857,  52.830914,  53.76597 ,  54.701026,  55.636081,
            56.571136,  57.50619 ,  58.441244,  59.376297,  60.311349,  61.246401,
            62.181452,  63.116502,  64.051551,  64.986599,  65.921645,  66.856691,
            67.791734,  68.726776,  69.661816,  70.596854,  71.531889,  72.466921,
            73.401949,  74.336973,  75.271992,  76.207005,  77.142011,  78.077007,
            79.011992,  79.946962,  80.881913,  81.816839,  82.751728,  83.686567,
            84.621327,  85.55596 ,  86.490367,  87.424304,  88.357004,  89.284228])

  • lon
    (lon)
    float64
    0.0 0.9375 1.875 ... 358.1 359.1

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.    ,   0.9375,   1.875 , ..., 357.1875, 358.125 , 359.0625])

• Data variables: (10)
  • time_bnds
    (time, bnds)
    datetime64[ns]
    dask.array<chunksize=(1826, 2), meta=np.ndarray>

    Array Chunk Bytes 490.80 kiB 28.55 kiB Shape (31411, 2) (1827, 2) Dask graph 18 chunks in 37 graph layers Data type datetime64[ns] numpy.ndarray

  • lev_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    formula :

    p = ap + b*ps

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    units :

    1

    formula_terms :

    ap: ap_bnds b: b_bnds ps: ps

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ap
    (time, lev)
    float64
    dask.array<chunksize=(1826, 95), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k)

    units :

    Pa

    Array Chunk Bytes 22.77 MiB 1.32 MiB Shape (31411, 95) (1827, 95) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b
    (time, lev)
    float64
    dask.array<chunksize=(1826, 95), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k)

    Array Chunk Bytes 22.77 MiB 1.32 MiB Shape (31411, 95) (1827, 95) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ps
    (time, lat, lon)
    float32
    dask.array<chunksize=(1826, 192, 384), meta=np.ndarray>

    long_name :

    Surface Air Pressure

    units :

    Pa

    Array Chunk Bytes 8.63 GiB 513.84 MiB Shape (31411, 192, 384) (1827, 192, 384) Dask graph 18 chunks in 37 graph layers Data type float32 numpy.ndarray

  • ap_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k+1/2)

    units :

    Pa

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 95, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k+1/2)

    Array Chunk Bytes 45.53 MiB 2.65 MiB Shape (31411, 95, 2) (1827, 95, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>

    Array Chunk Bytes 92.02 MiB 5.35 MiB Shape (31411, 192, 2) (1827, 192, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>

    Array Chunk Bytes 184.05 MiB 10.71 MiB Shape (31411, 384, 2) (1827, 384, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ta
    (time, lev, lat, lon)
    float32
    dask.array<chunksize=(1826, 95, 192, 384), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Air Temperature

    comment :

    Air Temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-20T10:06:39Z altered by CMOR: Reordered dimensions, original order: time lat lon lev. 2019-07-20T10:06:39Z altered by CMOR: replaced missing value flag (-9e+33) with standard missing value (1e+20). 2019-07-20T10:06:40Z altered by CMOR: Converted type from 'd' to 'f'. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lev. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lat.

    Array Chunk Bytes 819.59 GiB 47.67 GiB Shape (31411, 95, 192, 384) (1827, 95, 192, 384) Dask graph 18 chunks in 37 graph layers Data type float32 numpy.ndarray

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-01-01 12:00:00', '2015-01-02 12:00:00',
                   '2015-01-03 12:00:00', '2015-01-04 12:00:00',
                   '2015-01-05 12:00:00', '2015-01-06 12:00:00',
                   '2015-01-07 12:00:00', '2015-01-08 12:00:00',
                   '2015-01-09 12:00:00', '2015-01-10 12:00:00',
                   ...
                   '2100-12-22 12:00:00', '2100-12-23 12:00:00',
                   '2100-12-24 12:00:00', '2100-12-25 12:00:00',
                   '2100-12-26 12:00:00', '2100-12-27 12:00:00',
                   '2100-12-28 12:00:00', '2100-12-29 12:00:00',
                   '2100-12-30 12:00:00', '2100-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='time', length=31411, freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([    0.9961499869823456,     0.9826499819755554,     0.9589555908567136,
               0.9276459693502798,     0.8908393983675643,     0.8501502852051046,
               0.8068474689019394,     0.7620136942249918,     0.7164562136862798,
               0.6707187556893979,     0.6252446440590949,     0.5803910498630256,
               0.5366073683506467,     0.4942692315672581,     0.4537200746596845,
               0.4154341220199812,     0.3797068559311656,     0.3466272163038162,
              0.31615186284801977,     0.2881811466818235,     0.2625505749313972,
               0.2391083761673929,     0.2177224959069833,    0.19823367227649222,
              0.18046804676613204,     0.1642866912611011,     0.1495502438573978,
               0.1361203084601691,    0.12389576315665891,     0.1127819063778702,
              0.10265254219114787,    0.09340926514608601,    0.08499143437525487,
              0.07732509098198866,    0.07033859949420182,    0.06396105122748581,
              0.05814545167160128,   0.052848372347643724,    0.04802119957438934,
             0.043620851838144584,    0.03960959667221811,   0.035953036331112756,
              0.03261979576278374,    0.02958129009375771,   0.026811458381522945,
             0.024286543147051566,    0.02198488918316679,    0.01988675441031335,
              0.01797414241765359,   0.016230649461047374,   0.014642066833209968,
             0.013196053174346163,   0.011881106828660868,   0.010686522473707747,
             0.009602346544997533,   0.008619337479567604,   0.007728925959551567,
             0.006923178769969776,   0.006194759042298915,    0.00553688770278189,
             0.004943309739698989,    0.00440826408485998,   0.003926448676134962,
            0.0034929879294658276,  0.0031034074393196396,   0.002753606269661362,
             0.002439826760149696,   0.002158630883108384,  0.0019068793864594898,
            0.0016817067200655764,  0.0014805012328107652,  0.0013008852943855245,
            0.0011406967724043146,  0.0009979718648560865,  0.0008709289114062885,
            0.0007579534847354583,  0.0006575843101367814,  0.0005685002392614267,
            0.0004895081274191186, 0.00041953176502648504,  0.0003577273067885671,
           0.00030342777834802155, 0.00025597185256773534, 0.00021471303567573402,
            0.0001790279498548529, 0.00014832281342795124, 0.00012203819864095655,
            9.965253293940734e-05,  8.068427314824185e-05,  6.469292667745632e-05,
            5.127906093413877e-05, 4.0083522123670446e-05,  3.078596226644904e-05,
           2.3102853258774366e-05,  9.815852317932415e-06],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([-89.28422753251364, -88.35700351866494, -87.42430374606988,
           -86.49036676628116, -85.55596048489265,  -84.6213271076488,
           -83.68656681656385, -82.75172847343066, -81.81683872860319,
            -80.8819133467975,
           ...
             80.8819133467975,  81.81683872860319,  82.75172847343066,
            83.68656681656385,   84.6213271076488,  85.55596048489265,
            86.49036676628116,  87.42430374606988,  88.35700351866494,
            89.28422753251364],
          dtype='float64', name='lat', length=192))

  • lon
    PandasIndex

    PandasIndex(Index([     0.0,   0.9375,    1.875,   2.8125,     3.75,   4.6875,    5.625,
             6.5625,      7.5,   8.4375,
           ...
            350.625, 351.5625,    352.5, 353.4375,  354.375, 355.3125,   356.25,
           357.1875,  358.125, 359.0625],
          dtype='float64', name='lon', length=384))

• Attributes: (47)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  ScenarioMIP

  branch_method :

  standard

  branch_time_in_child :

  60265.0

  branch_time_in_parent :

  60265.0

  contact :

  cmip6-mpi-esm@dkrz.de

  creation_date :

  2019-07-20T10:06:40Z

  data_specs_version :

  01.00.30

  experiment :

  update of RCP2.6 based on SSP1

  experiment_id :

  ssp126

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  day

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.DKRZ.MPI-ESM1-2-HR.ssp126.none.r1i1p1f1

  grid :

  gn

  grid_label :

  gn

  history :

  2019-07-20T10:06:40Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.

  initialization_index :

  1

  institution :

  Deutsches Klimarechenzentrum, Hamburg 20146, Germany

  institution_id :

  DKRZ

  mip_era :

  CMIP6

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  historical

  parent_mip_era :

  CMIP6

  parent_source_id :

  MPI-ESM1-2-HR

  parent_time_units :

  days since 1850-1-1 00:00:00

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  project_id :

  CMIP6

  realization_index :

  1

  realm :

  atmos

  references :

  MPI-ESM: Mauritsen, T. et al. (2019), Developments in the MPI‐M Earth System Model version 1.2 (MPI‐ESM1.2) and Its Response to Increasing CO2, J. Adv. Model. Earth Syst.,11, 998-1038, doi:10.1029/2018MS001400, Mueller, W.A. et al. (2018): A high‐resolution version of the Max Planck Institute Earth System Model MPI‐ESM1.2‐HR. J. Adv. Model. EarthSyst.,10,1383–1413, doi:10.1029/2017MS001217

  source :

  MPI-ESM1.2-HR (2017): aerosol: none, prescribed MACv2-SP atmos: ECHAM6.3 (spectral T127; 384 x 192 longitude/latitude; 95 levels; top level 0.01 hPa) atmosChem: none land: JSBACH3.20 landIce: none/prescribed ocean: MPIOM1.63 (tripolar TP04, approximately 0.4deg; 802 x 404 longitude/latitude; 40 levels; top grid cell 0-12 m) ocnBgchem: HAMOCC6 seaIce: unnamed (thermodynamic (Semtner zero-layer) dynamic (Hibler 79) sea ice model)

  source_id :

  MPI-ESM1-2-HR

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  CFday

  table_info :

  Creation Date:(09 May 2019) MD5:e6ef8ececc8f338646ebfb3aeed36bfc

  title :

  MPI-ESM1-2-HR output prepared for CMIP6

  variable_id :

  ta

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by DKRZ is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

For this case, this is the worst configuration you can have in dask because the chunk size most likely exceeds the available memory so that you cannot really work with this dataset at all.

Tip

When using open_mfdataset, ensure that the chunks are well defined. Consider reopening with a multiple of the storage chunk sizes.

open_chunks = dict(
    time=2,
    lev=storage_chunks[1],  # storage chunk levels,
    lon=len(one_file_per_chunk["lon"]),
    lat=len(one_file_per_chunk["lat"]),
)
multiple_storage_chunks = xr.open_mfdataset(
    "/work/ik1017/CMIP6/data/CMIP6/ScenarioMIP/DKRZ/MPI-ESM1-2-HR/ssp126/r1i1p1f1/CFday/ta/gn/v20190710/*.nc",
    chunks=open_chunks,  # or a multiple storage_chunks
)
multiple_storage_chunks

<xarray.Dataset>
Dimensions:    (time: 31411, bnds: 2, lev: 95, lat: 192, lon: 384)
Coordinates:
  * time       (time) datetime64[ns] 2015-01-01T12:00:00 ... 2100-12-31T12:00:00
  * lev        (lev) float64 0.9961 0.9826 0.959 ... 2.31e-05 9.816e-06
  * lat        (lat) float64 -89.28 -88.36 -87.42 -86.49 ... 87.42 88.36 89.28
  * lon        (lon) float64 0.0 0.9375 1.875 2.812 ... 356.2 357.2 358.1 359.1
Dimensions without coordinates: bnds
Data variables:
    time_bnds  (time, bnds) datetime64[ns] dask.array<chunksize=(2, 2), meta=np.ndarray>
    lev_bnds   (time, lev, bnds) float64 dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>
    ap         (time, lev) float64 dask.array<chunksize=(1826, 48), meta=np.ndarray>
    b          (time, lev) float64 dask.array<chunksize=(1826, 48), meta=np.ndarray>
    ps         (time, lat, lon) float32 dask.array<chunksize=(2, 192, 384), meta=np.ndarray>
    ap_bnds    (time, lev, bnds) float64 dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>
    b_bnds     (time, lev, bnds) float64 dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>
    lat_bnds   (time, lat, bnds) float64 dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>
    lon_bnds   (time, lon, bnds) float64 dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>
    ta         (time, lev, lat, lon) float32 dask.array<chunksize=(2, 48, 192, 384), meta=np.ndarray>
Attributes: (12/47)
    Conventions:            CF-1.7 CMIP-6.2
    activity_id:            ScenarioMIP
    branch_method:          standard
    branch_time_in_child:   60265.0
    branch_time_in_parent:  60265.0
    contact:                cmip6-mpi-esm@dkrz.de
    ...                     ...
    title:                  MPI-ESM1-2-HR output prepared for CMIP6
    variable_id:            ta
    variant_label:          r1i1p1f1
    license:                CMIP6 model data produced by DKRZ is licensed und...
    cmor_version:           3.4.0
    tracking_id:            hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

xarray.Dataset

• Dimensions:
  • time: 31411
  • bnds: 2
  • lev: 95
  • lat: 192
  • lon: 384
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2015-01-01T12:00:00 ... 2100-12-...

    bounds :

    time_bnds

    axis :

    T

    long_name :

    time

    standard_name :

    time

    array(['2015-01-01T12:00:00.000000000', '2015-01-02T12:00:00.000000000',
           '2015-01-03T12:00:00.000000000', ..., '2100-12-29T12:00:00.000000000',
           '2100-12-30T12:00:00.000000000', '2100-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9961 0.9826 ... 9.816e-06

    bounds :

    lev_bnds

    units :

    1

    axis :

    Z

    positive :

    down

    long_name :

    hybrid sigma pressure coordinate

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula :

    p = ap + b*ps

    formula_terms :

    ap: ap b: b ps: ps

    array([9.961500e-01, 9.826500e-01, 9.589556e-01, 9.276460e-01, 8.908394e-01,
           8.501503e-01, 8.068475e-01, 7.620137e-01, 7.164562e-01, 6.707188e-01,
           6.252446e-01, 5.803910e-01, 5.366074e-01, 4.942692e-01, 4.537201e-01,
           4.154341e-01, 3.797069e-01, 3.466272e-01, 3.161519e-01, 2.881811e-01,
           2.625506e-01, 2.391084e-01, 2.177225e-01, 1.982337e-01, 1.804680e-01,
           1.642867e-01, 1.495502e-01, 1.361203e-01, 1.238958e-01, 1.127819e-01,
           1.026525e-01, 9.340927e-02, 8.499143e-02, 7.732509e-02, 7.033860e-02,
           6.396105e-02, 5.814545e-02, 5.284837e-02, 4.802120e-02, 4.362085e-02,
           3.960960e-02, 3.595304e-02, 3.261980e-02, 2.958129e-02, 2.681146e-02,
           2.428654e-02, 2.198489e-02, 1.988675e-02, 1.797414e-02, 1.623065e-02,
           1.464207e-02, 1.319605e-02, 1.188111e-02, 1.068652e-02, 9.602347e-03,
           8.619337e-03, 7.728926e-03, 6.923179e-03, 6.194759e-03, 5.536888e-03,
           4.943310e-03, 4.408264e-03, 3.926449e-03, 3.492988e-03, 3.103407e-03,
           2.753606e-03, 2.439827e-03, 2.158631e-03, 1.906879e-03, 1.681707e-03,
           1.480501e-03, 1.300885e-03, 1.140697e-03, 9.979719e-04, 8.709289e-04,
           7.579535e-04, 6.575843e-04, 5.685002e-04, 4.895081e-04, 4.195318e-04,
           3.577273e-04, 3.034278e-04, 2.559719e-04, 2.147130e-04, 1.790279e-04,
           1.483228e-04, 1.220382e-04, 9.965253e-05, 8.068427e-05, 6.469293e-05,
           5.127906e-05, 4.008352e-05, 3.078596e-05, 2.310285e-05, 9.815852e-06])

  • lat
    (lat)
    float64
    -89.28 -88.36 ... 88.36 89.28

    bounds :

    lat_bnds

    units :

    degrees_north

    axis :

    Y

    long_name :

    Latitude

    standard_name :

    latitude

    array([-89.284228, -88.357004, -87.424304, -86.490367, -85.55596 , -84.621327,
           -83.686567, -82.751728, -81.816839, -80.881913, -79.946962, -79.011992,
           -78.077007, -77.142011, -76.207005, -75.271992, -74.336973, -73.401949,
           -72.466921, -71.531889, -70.596854, -69.661816, -68.726776, -67.791734,
           -66.856691, -65.921645, -64.986599, -64.051551, -63.116502, -62.181452,
           -61.246401, -60.311349, -59.376297, -58.441244, -57.50619 , -56.571136,
           -55.636081, -54.701026, -53.76597 , -52.830914, -51.895857, -50.960801,
           -50.025743, -49.090686, -48.155628, -47.22057 , -46.285512, -45.350454,
           -44.415395, -43.480336, -42.545277, -41.610218, -40.675159, -39.740099,
           -38.805039, -37.86998 , -36.93492 , -35.99986 , -35.064799, -34.129739,
           -33.194679, -32.259618, -31.324558, -30.389497, -29.454436, -28.519375,
           -27.584315, -26.649254, -25.714193, -24.779132, -23.84407 , -22.909009,
           -21.973948, -21.038887, -20.103825, -19.168764, -18.233703, -17.298641,
           -16.36358 , -15.428518, -14.493457, -13.558395, -12.623333, -11.688272,
           -10.75321 ,  -9.818148,  -8.883087,  -7.948025,  -7.012963,  -6.077901,
            -5.14284 ,  -4.207778,  -3.272716,  -2.337654,  -1.402593,  -0.467531,
             0.467531,   1.402593,   2.337654,   3.272716,   4.207778,   5.14284 ,
             6.077901,   7.012963,   7.948025,   8.883087,   9.818148,  10.75321 ,
            11.688272,  12.623333,  13.558395,  14.493457,  15.428518,  16.36358 ,
            17.298641,  18.233703,  19.168764,  20.103825,  21.038887,  21.973948,
            22.909009,  23.84407 ,  24.779132,  25.714193,  26.649254,  27.584315,
            28.519375,  29.454436,  30.389497,  31.324558,  32.259618,  33.194679,
            34.129739,  35.064799,  35.99986 ,  36.93492 ,  37.86998 ,  38.805039,
            39.740099,  40.675159,  41.610218,  42.545277,  43.480336,  44.415395,
            45.350454,  46.285512,  47.22057 ,  48.155628,  49.090686,  50.025743,
            50.960801,  51.895857,  52.830914,  53.76597 ,  54.701026,  55.636081,
            56.571136,  57.50619 ,  58.441244,  59.376297,  60.311349,  61.246401,
            62.181452,  63.116502,  64.051551,  64.986599,  65.921645,  66.856691,
            67.791734,  68.726776,  69.661816,  70.596854,  71.531889,  72.466921,
            73.401949,  74.336973,  75.271992,  76.207005,  77.142011,  78.077007,
            79.011992,  79.946962,  80.881913,  81.816839,  82.751728,  83.686567,
            84.621327,  85.55596 ,  86.490367,  87.424304,  88.357004,  89.284228])

  • lon
    (lon)
    float64
    0.0 0.9375 1.875 ... 358.1 359.1

    bounds :

    lon_bnds

    units :

    degrees_east

    axis :

    X

    long_name :

    Longitude

    standard_name :

    longitude

    array([  0.    ,   0.9375,   1.875 , ..., 357.1875, 358.125 , 359.0625])

• Data variables: (10)
  • time_bnds
    (time, bnds)
    datetime64[ns]
    dask.array<chunksize=(2, 2), meta=np.ndarray>

    Array Chunk Bytes 490.80 kiB 32 B Shape (31411, 2) (2, 2) Dask graph 15708 chunks in 37 graph layers Data type datetime64[ns] numpy.ndarray

  • lev_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>

    formula :

    p = ap + b*ps

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    units :

    1

    formula_terms :

    ap: ap_bnds b: b_bnds ps: ps

    Array Chunk Bytes 45.53 MiB 1.34 MiB Shape (31411, 95, 2) (1827, 48, 2) Dask graph 36 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ap
    (time, lev)
    float64
    dask.array<chunksize=(1826, 48), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k)

    units :

    Pa

    Array Chunk Bytes 22.77 MiB 685.12 kiB Shape (31411, 95) (1827, 48) Dask graph 36 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b
    (time, lev)
    float64
    dask.array<chunksize=(1826, 48), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k)

    Array Chunk Bytes 22.77 MiB 685.12 kiB Shape (31411, 95) (1827, 48) Dask graph 36 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ps
    (time, lat, lon)
    float32
    dask.array<chunksize=(2, 192, 384), meta=np.ndarray>

    long_name :

    Surface Air Pressure

    units :

    Pa

    Array Chunk Bytes 8.63 GiB 576.00 kiB Shape (31411, 192, 384) (2, 192, 384) Dask graph 15708 chunks in 37 graph layers Data type float32 numpy.ndarray

  • ap_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: ap(k+1/2)

    units :

    Pa

    Array Chunk Bytes 45.53 MiB 1.34 MiB Shape (31411, 95, 2) (1827, 48, 2) Dask graph 36 chunks in 55 graph layers Data type float64 numpy.ndarray

  • b_bnds
    (time, lev, bnds)
    float64
    dask.array<chunksize=(1826, 48, 2), meta=np.ndarray>

    long_name :

    vertical coordinate formula term: b(k+1/2)

    Array Chunk Bytes 45.53 MiB 1.34 MiB Shape (31411, 95, 2) (1827, 48, 2) Dask graph 36 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lat_bnds
    (time, lat, bnds)
    float64
    dask.array<chunksize=(1826, 192, 2), meta=np.ndarray>

    Array Chunk Bytes 92.02 MiB 5.35 MiB Shape (31411, 192, 2) (1827, 192, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • lon_bnds
    (time, lon, bnds)
    float64
    dask.array<chunksize=(1826, 384, 2), meta=np.ndarray>

    Array Chunk Bytes 184.05 MiB 10.71 MiB Shape (31411, 384, 2) (1827, 384, 2) Dask graph 18 chunks in 55 graph layers Data type float64 numpy.ndarray

  • ta
    (time, lev, lat, lon)
    float32
    dask.array<chunksize=(2, 48, 192, 384), meta=np.ndarray>

    standard_name :

    air_temperature

    long_name :

    Air Temperature

    comment :

    Air Temperature

    units :

    K

    cell_methods :

    area: time: mean

    cell_measures :

    area: areacella

    history :

    2019-07-20T10:06:39Z altered by CMOR: Reordered dimensions, original order: time lat lon lev. 2019-07-20T10:06:39Z altered by CMOR: replaced missing value flag (-9e+33) with standard missing value (1e+20). 2019-07-20T10:06:40Z altered by CMOR: Converted type from 'd' to 'f'. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lev. 2019-07-20T10:06:41Z altered by CMOR: Inverted axis: lat.

    Array Chunk Bytes 819.59 GiB 27.00 MiB Shape (31411, 95, 192, 384) (2, 48, 192, 384) Dask graph 31416 chunks in 37 graph layers Data type float32 numpy.ndarray

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2015-01-01 12:00:00', '2015-01-02 12:00:00',
                   '2015-01-03 12:00:00', '2015-01-04 12:00:00',
                   '2015-01-05 12:00:00', '2015-01-06 12:00:00',
                   '2015-01-07 12:00:00', '2015-01-08 12:00:00',
                   '2015-01-09 12:00:00', '2015-01-10 12:00:00',
                   ...
                   '2100-12-22 12:00:00', '2100-12-23 12:00:00',
                   '2100-12-24 12:00:00', '2100-12-25 12:00:00',
                   '2100-12-26 12:00:00', '2100-12-27 12:00:00',
                   '2100-12-28 12:00:00', '2100-12-29 12:00:00',
                   '2100-12-30 12:00:00', '2100-12-31 12:00:00'],
                  dtype='datetime64[ns]', name='time', length=31411, freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([    0.9961499869823456,     0.9826499819755554,     0.9589555908567136,
               0.9276459693502798,     0.8908393983675643,     0.8501502852051046,
               0.8068474689019394,     0.7620136942249918,     0.7164562136862798,
               0.6707187556893979,     0.6252446440590949,     0.5803910498630256,
               0.5366073683506467,     0.4942692315672581,     0.4537200746596845,
               0.4154341220199812,     0.3797068559311656,     0.3466272163038162,
              0.31615186284801977,     0.2881811466818235,     0.2625505749313972,
               0.2391083761673929,     0.2177224959069833,    0.19823367227649222,
              0.18046804676613204,     0.1642866912611011,     0.1495502438573978,
               0.1361203084601691,    0.12389576315665891,     0.1127819063778702,
              0.10265254219114787,    0.09340926514608601,    0.08499143437525487,
              0.07732509098198866,    0.07033859949420182,    0.06396105122748581,
              0.05814545167160128,   0.052848372347643724,    0.04802119957438934,
             0.043620851838144584,    0.03960959667221811,   0.035953036331112756,
              0.03261979576278374,    0.02958129009375771,   0.026811458381522945,
             0.024286543147051566,    0.02198488918316679,    0.01988675441031335,
              0.01797414241765359,   0.016230649461047374,   0.014642066833209968,
             0.013196053174346163,   0.011881106828660868,   0.010686522473707747,
             0.009602346544997533,   0.008619337479567604,   0.007728925959551567,
             0.006923178769969776,   0.006194759042298915,    0.00553688770278189,
             0.004943309739698989,    0.00440826408485998,   0.003926448676134962,
            0.0034929879294658276,  0.0031034074393196396,   0.002753606269661362,
             0.002439826760149696,   0.002158630883108384,  0.0019068793864594898,
            0.0016817067200655764,  0.0014805012328107652,  0.0013008852943855245,
            0.0011406967724043146,  0.0009979718648560865,  0.0008709289114062885,
            0.0007579534847354583,  0.0006575843101367814,  0.0005685002392614267,
            0.0004895081274191186, 0.00041953176502648504,  0.0003577273067885671,
           0.00030342777834802155, 0.00025597185256773534, 0.00021471303567573402,
            0.0001790279498548529, 0.00014832281342795124, 0.00012203819864095655,
            9.965253293940734e-05,  8.068427314824185e-05,  6.469292667745632e-05,
            5.127906093413877e-05, 4.0083522123670446e-05,  3.078596226644904e-05,
           2.3102853258774366e-05,  9.815852317932415e-06],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([-89.28422753251364, -88.35700351866494, -87.42430374606988,
           -86.49036676628116, -85.55596048489265,  -84.6213271076488,
           -83.68656681656385, -82.75172847343066, -81.81683872860319,
            -80.8819133467975,
           ...
             80.8819133467975,  81.81683872860319,  82.75172847343066,
            83.68656681656385,   84.6213271076488,  85.55596048489265,
            86.49036676628116,  87.42430374606988,  88.35700351866494,
            89.28422753251364],
          dtype='float64', name='lat', length=192))

  • lon
    PandasIndex

    PandasIndex(Index([     0.0,   0.9375,    1.875,   2.8125,     3.75,   4.6875,    5.625,
             6.5625,      7.5,   8.4375,
           ...
            350.625, 351.5625,    352.5, 353.4375,  354.375, 355.3125,   356.25,
           357.1875,  358.125, 359.0625],
          dtype='float64', name='lon', length=384))

• Attributes: (47)

  Conventions :

  CF-1.7 CMIP-6.2

  activity_id :

  ScenarioMIP

  branch_method :

  standard

  branch_time_in_child :

  60265.0

  branch_time_in_parent :

  60265.0

  contact :

  cmip6-mpi-esm@dkrz.de

  creation_date :

  2019-07-20T10:06:40Z

  data_specs_version :

  01.00.30

  experiment :

  update of RCP2.6 based on SSP1

  experiment_id :

  ssp126

  external_variables :

  areacella

  forcing_index :

  1

  frequency :

  day

  further_info_url :

  https://furtherinfo.es-doc.org/CMIP6.DKRZ.MPI-ESM1-2-HR.ssp126.none.r1i1p1f1

  grid :

  gn

  grid_label :

  gn

  history :

  2019-07-20T10:06:40Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.

  initialization_index :

  1

  institution :

  Deutsches Klimarechenzentrum, Hamburg 20146, Germany

  institution_id :

  DKRZ

  mip_era :

  CMIP6

  nominal_resolution :

  100 km

  parent_activity_id :

  CMIP

  parent_experiment_id :

  historical

  parent_mip_era :

  CMIP6

  parent_source_id :

  MPI-ESM1-2-HR

  parent_time_units :

  days since 1850-1-1 00:00:00

  parent_variant_label :

  r1i1p1f1

  physics_index :

  1

  product :

  model-output

  project_id :

  CMIP6

  realization_index :

  1

  realm :

  atmos

  references :

  MPI-ESM: Mauritsen, T. et al. (2019), Developments in the MPI‐M Earth System Model version 1.2 (MPI‐ESM1.2) and Its Response to Increasing CO2, J. Adv. Model. Earth Syst.,11, 998-1038, doi:10.1029/2018MS001400, Mueller, W.A. et al. (2018): A high‐resolution version of the Max Planck Institute Earth System Model MPI‐ESM1.2‐HR. J. Adv. Model. EarthSyst.,10,1383–1413, doi:10.1029/2017MS001217

  source :

  MPI-ESM1.2-HR (2017): aerosol: none, prescribed MACv2-SP atmos: ECHAM6.3 (spectral T127; 384 x 192 longitude/latitude; 95 levels; top level 0.01 hPa) atmosChem: none land: JSBACH3.20 landIce: none/prescribed ocean: MPIOM1.63 (tripolar TP04, approximately 0.4deg; 802 x 404 longitude/latitude; 40 levels; top grid cell 0-12 m) ocnBgchem: HAMOCC6 seaIce: unnamed (thermodynamic (Semtner zero-layer) dynamic (Hibler 79) sea ice model)

  source_id :

  MPI-ESM1-2-HR

  source_type :

  AOGCM

  sub_experiment :

  none

  sub_experiment_id :

  none

  table_id :

  CFday

  table_info :

  Creation Date:(09 May 2019) MD5:e6ef8ececc8f338646ebfb3aeed36bfc

  title :

  MPI-ESM1-2-HR output prepared for CMIP6

  variable_id :

  ta

  variant_label :

  r1i1p1f1

  license :

  CMIP6 model data produced by DKRZ is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.

  cmor_version :

  3.4.0

  tracking_id :

  hdl:21.14100/c30b8347-4287-4f20-ab6f-d8ddb1198635

Best-practices

In the following, we are running some example analysis to explore how dask works. For ds_auto, we want to know:

What is the quick and unweighted field mean of the first year?

def calc_fieldmean_of_first_year(ds):
    dsy1 = list(dict(ds.groupby("time.year")).values())[0]
    dsy1mean = dsy1.mean(dim="cell")
    display(dsy1mean)
    gbs = dsy1.nbytes / 1024 / 1024 / 1024
    print(f"Going to process: {gbs} GBs")
    return dsy1mean.compute(), gbs

Do we need dask?

print(
    "The first year of the dataset is "
    f"{ds_delayed.isel(time=slice(0,2*24*365)).nbytes/1024/1024/1024}"
    " GBs large and does not fit into memory"
)

The first year of the dataset is 821.25 GBs large and does not fit into memory

# ds_delayed.isel(time=slice(0,2*24*365)).mean(dim="time").compute()

Let’s quickly define a client to get a dashboard view. More on this in the distributed notebook.

# client.close()
client = Client()

Diagnostic tools

When we let dask actually work, we can use diagnostic-tools to inspect how our task-graph was evaluated from the client i.e. how performant the execution was. E.g., there is the performance_report. We use that in a with context as follows:

s = time.time()
with performance_report(filename="dask-report-auto.html"):
    result, gbs = calc_fieldmean_of_first_year(ds_auto)
e = time.time()
print(f"Process speed: {gbs/(e-s)} GB/s")

<xarray.DataArray 'pr' (time: 16655)>
dask.array<mean_agg-aggregate, shape=(16655,), dtype=float32, chunksize=(32,), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2020-12-31T23:30:00

xarray.DataArray

'pr'

• time: 16655

• dask.array<chunksize=(32,), meta=np.ndarray>

  Array Chunk Bytes 65.06 kiB 128 B Shape (16655,) (32,) Dask graph 521 chunks in 7 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2020-12-...

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2020-12-31T22:30:00.000000000',
           '2020-12-31T23:00:00.000000000', '2020-12-31T23:30:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2020-12-31 19:00:00', '2020-12-31 19:30:00',
                   '2020-12-31 20:00:00', '2020-12-31 20:30:00',
                   '2020-12-31 21:00:00', '2020-12-31 21:30:00',
                   '2020-12-31 22:00:00', '2020-12-31 22:30:00',
                   '2020-12-31 23:00:00', '2020-12-31 23:30:00'],
                  dtype='datetime64[ns]', name='time', length=16655, freq=None))

• Attributes: (0)

Going to process: 780.703125 GBs
Process speed: 6.697790692183614 GB/s

Lets compare this with the dataset with storage chunk dask chunks:

s = time.time()
with performance_report(filename="dask-report-storage.html"):
    gbs = calc_fieldmean_of_first_year(ds_storage)[1]
e = time.time()
print(f"Process speed: {gbs/(e-s)} GB/s")

<xarray.DataArray 'pr' (time: 16655)>
dask.array<mean_agg-aggregate, shape=(16655,), dtype=float32, chunksize=(16,), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2020-12-31T23:30:00

xarray.DataArray

'pr'

• time: 16655

• dask.array<chunksize=(16,), meta=np.ndarray>

  Array Chunk Bytes 65.06 kiB 64 B Shape (16655,) (16,) Dask graph 1041 chunks in 7 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2020-12-...

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2020-12-31T22:30:00.000000000',
           '2020-12-31T23:00:00.000000000', '2020-12-31T23:30:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2020-12-31 19:00:00', '2020-12-31 19:30:00',
                   '2020-12-31 20:00:00', '2020-12-31 20:30:00',
                   '2020-12-31 21:00:00', '2020-12-31 21:30:00',
                   '2020-12-31 22:00:00', '2020-12-31 22:30:00',
                   '2020-12-31 23:00:00', '2020-12-31 23:30:00'],
                  dtype='datetime64[ns]', name='time', length=16655, freq=None))

• Attributes: (0)

Going to process: 780.703125 GBs

/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/distributed/client.py:3106: UserWarning: Sending large graph of size 9.65 MiB.
This may cause some slowdown.
Consider scattering data ahead of time and using futures.
  warnings.warn(

Process speed: 6.211303828719483 GB/s

We found that the field mean of the 780GBs of the first year was processed for

• dask_auto:

  • in *521 chunks in 7 graph layers

  • with a speed of about 6.5GB/s (10GB/s can be considered max for a levante node)

• dask_storage:

  • in *1041 chunks in 7 graph layers

  • with a speed of about 5.5GB/s

result.hvplot.line()

Adapt chunks to your analysis

Now we ask

What is the histogram for the first 8000 cells?

The analysis goal has implications for our chunk setting. One dask chunk should contain

• 8000 cells

• as many timesteps as possible

Hoever, the general optimization rules for dask chunks setting say:

• do not split up storage chunks

and

• target 100MB dask chunks

So we can come up with sth like

chunked_for_histogram = xr.open_dataset(
    f"/fastdata/ka1081/nextgems_cycle3/ngc3028/ngc3028_{selection['time']}_{str(selection['zoom'])}.zarr",
    engine="zarr",
    chunks=dict(time=16 * 8, cell=262144),
)[VARID]
chunked_for_histogram

<xarray.DataArray 'pr' (time: 96480, cell: 12582912)>
dask.array<open_dataset-76e6ce1b4c49b5e250a66fdc64b44210pr, shape=(96480, 12582912), dtype=float32, chunksize=(128, 262144), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 12582912

• dask.array<chunksize=(128, 262144), meta=np.ndarray>

  Array Chunk Bytes 4.42 TiB 128.00 MiB Shape (96480, 12582912) (128, 262144) Dask graph 36192 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

Lets subselect

dss = ds_auto.isel(cell=slice(0, 8000))
dss

<xarray.DataArray 'pr' (time: 96480, cell: 8000)>
dask.array<getitem, shape=(96480, 8000), dtype=float32, chunksize=(32, 8000), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 8000

• dask.array<chunksize=(32, 8000), meta=np.ndarray>

  Array Chunk Bytes 2.88 GiB 0.98 MiB Shape (96480, 8000) (32, 8000) Dask graph 3015 chunks in 3 graph layers Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

Persist for more analysis, compute for results

We can see that dss would fit into our memory. If we plan to run multiple different functions on it after loading, we can use .persist instead of compute. This keeps the data distributed across the workers so that a new dask job on the data is faster.

You can see in the dashboard that .persist will trigger some getitem() tasks.

dss_persist = dss.persist()
dss_persist

<xarray.DataArray 'pr' (time: 96480, cell: 8000)>
dask.array<getitem, shape=(96480, 8000), dtype=float32, chunksize=(32, 8000), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 8000

• dask.array<chunksize=(32, 8000), meta=np.ndarray>

  Array Chunk Bytes 2.88 GiB 0.98 MiB Shape (96480, 8000) (32, 8000) Dask graph 3015 chunks in 1 graph layer Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

dss_persist is still a dask array but a .load would return a numpy array very quickly by gathering the data from workers.

dss_persist

<xarray.DataArray 'pr' (time: 96480, cell: 8000)>
dask.array<getitem, shape=(96480, 8000), dtype=float32, chunksize=(32, 8000), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-01-20T00:30:00 ... 2025-07-22
Dimensions without coordinates: cell
Attributes:
    cell_methods:  time: mean
    component:     atmo
    grid_mapping:  crs
    long_name:     precipitation flux
    units:         kg m-2 s-1
    vgrid:         surface

xarray.DataArray

'pr'

• time: 96480
• cell: 8000

• dask.array<chunksize=(32, 8000), meta=np.ndarray>

  Array Chunk Bytes 2.88 GiB 0.98 MiB Shape (96480, 8000) (32, 8000) Dask graph 3015 chunks in 1 graph layer Data type float32 numpy.ndarray

• Coordinates: (1)
  • time
    (time)
    datetime64[ns]
    2020-01-20T00:30:00 ... 2025-07-22

    axis :

    T

    array(['2020-01-20T00:30:00.000000000', '2020-01-20T01:00:00.000000000',
           '2020-01-20T01:30:00.000000000', ..., '2025-07-21T23:00:00.000000000',
           '2025-07-21T23:30:00.000000000', '2025-07-22T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-01-20 00:30:00', '2020-01-20 01:00:00',
                   '2020-01-20 01:30:00', '2020-01-20 02:00:00',
                   '2020-01-20 02:30:00', '2020-01-20 03:00:00',
                   '2020-01-20 03:30:00', '2020-01-20 04:00:00',
                   '2020-01-20 04:30:00', '2020-01-20 05:00:00',
                   ...
                   '2025-07-21 19:30:00', '2025-07-21 20:00:00',
                   '2025-07-21 20:30:00', '2025-07-21 21:00:00',
                   '2025-07-21 21:30:00', '2025-07-21 22:00:00',
                   '2025-07-21 22:30:00', '2025-07-21 23:00:00',
                   '2025-07-21 23:30:00', '2025-07-22 00:00:00'],
                  dtype='datetime64[ns]', name='time', length=96480, freq=None))

• Attributes: (6)

  cell_methods :

  time: mean

  component :

  atmo

  grid_mapping :

  crs

  long_name :

  precipitation flux

  units :

  kg m-2 s-1

  vgrid :

  surface

Multiple computes at once

Now, we create the bins for our histogram. If we need to run .compute multiple times, we can run that together on the scheduler with dask.compute:

# NOT:
# dsmin=dss.min().compute()
# dsmax=dss.max().compute()
# BETTER:
dsmin, dsmax = dask.compute(dss_persist.min(), dss_persist.max())
bins = np.linspace(dsmin, dsmax - (dsmax - dsmin) * 0.95, 20)

For the histogram we use xhistogram:

def calc_histogram_for_fieldslice(dshist):
    h_x = histogram(dshist, bins=bins, dim=["time"])
    return h_x.compute(), gbs

with performance_report(filename="dask-report-auto.html"):
    result_histogram, gbs = calc_histogram_for_fieldslice(dss_persist)
result_histogram

<xarray.DataArray 'histogram_pr' (cell: 8000, pr_bin: 19)>
array([[95998,   159,    73, ...,     2,     5,     4],
       [96011,   161,    60, ...,     4,     3,     4],
       [96061,   121,    67, ...,     2,     4,     2],
       ...,
       [96346,    25,    12, ...,     1,     3,     1],
       [96341,    32,    17, ...,     1,     1,     1],
       [96353,    18,    18, ...,     2,     1,     1]])
Coordinates:
  * pr_bin   (pr_bin) float64 0.0001068 0.0003205 ... 0.003739 0.003953
Dimensions without coordinates: cell

xarray.DataArray

'histogram_pr'

• cell: 8000
• pr_bin: 19

• 95998 159 73 38 27 23 20 14 12 15 7 10 7 ... 5 6 5 1 1 3 2 4 1 1 2 1 1

  array([[95998,   159,    73, ...,     2,     5,     4],
         [96011,   161,    60, ...,     4,     3,     4],
         [96061,   121,    67, ...,     2,     4,     2],
         ...,
         [96346,    25,    12, ...,     1,     3,     1],
         [96341,    32,    17, ...,     1,     1,     1],
         [96353,    18,    18, ...,     2,     1,     1]])

• Coordinates: (1)
  • pr_bin
    (pr_bin)
    float64
    0.0001068 0.0003205 ... 0.003953

    cell_methods :

    time: mean

    component :

    atmo

    grid_mapping :

    crs

    long_name :

    precipitation flux

    units :

    kg m-2 s-1

    vgrid :

    surface

    array([0.000107, 0.000321, 0.000534, 0.000748, 0.000962, 0.001175, 0.001389,
           0.001603, 0.001816, 0.00203 , 0.002244, 0.002457, 0.002671, 0.002885,
           0.003098, 0.003312, 0.003526, 0.003739, 0.003953])

• Indexes: (1)
  • pr_bin
    PandasIndex

    PandasIndex(Index([0.00010683618878063417,  0.0003205085663419025,  0.0005341809439031709,
            0.0007478533214644392,  0.0009615256990257075,   0.001175198076586976,
            0.0013888704541482441,  0.0016025428317095127,  0.0018162152092707808,
            0.0020298875868320493,   0.002243559964393318,   0.002457232341954586,
             0.002670904719515854,  0.0028845770970771225,   0.003098249474638391,
             0.003311921852199659,  0.0035255942297609277,  0.0037392666073221958,
             0.003952938984883464],
          dtype='float64', name='pr_bin'))

• Attributes: (0)

result_histogram.isel(cell=1, pr_bin=slice(1, 99)).hvplot.bar(x="pr_bin")