Finding and loading data with Intake

Intake is a powerful, flexible data cataloging system that allows you to discover, describe, and load datasets from diverse sources using a unified interface.

This tutorial walks you through the core workflow:

1. Opening a catalog from a URL

2. Exploring its contents

3. Loading a dataset with lazy evaluation and chunking

4. Using the data for analysis or visualization

We’ll use a public catalog from the nextGEMS project to demonstrate real-world usage.

Intake v2

Version 2 of Intake introduced significant, non-backward-compatible changes to the catalog specification. The catalogs that we maintain are still using the old format. Therefore, it is important to constrain the Intake package in your Python environment, e.g., intake<2 and intake-xarray<2.

Command-line use

See this tutorial for a little tool for querying the catalogs from the command line.

Opening the catalog

The first step is to open a catalog using intake.open_catalog(). This function loads a catalog file and returns a catalog object that you can explore and query.

In this example, we’re loading a public YAML catalog hosted online. The catalog contains metadata about datasets, including dataset locations, temporal resolution, and additional metadata.

import intake

cat = intake.open_catalog("https://data.nextgems-h2020.eu/online.yaml")

Online datasets

The online.yaml catalog contains datasets that are publicly accessible via web servers. This means that you can run the tutorials on any machine with an internet connection.

Exploring the catalog structure

Once the catalog is loaded, you can inspect its entries. Each entry corresponds to a dataset or sub-catalog.

list(cat)

['ICON', 'ERA5', 'JRA3Q', 'MERRA2', 'IFS', 'IMERG', 'GSMaP', 'tutorial']

You’ll see entries like ICON, IFS, or tutorial and others, each representing a dataset or group of datasets. You can access a specific entry directly via attribute access, e.g., cat.tutorial.

list(cat.tutorial)

['ICON.native.2d_PT6H_inst',
 'ICON.native.2d_P1D_mean',
 'ICON.native.3d_P1D_mean']

This shows all datasets in the tutorial sub-catalog. You can access individual datasets using their identifier to show the dataset description, parameters, and its driver (how it’s loaded).

cat.tutorial["ICON.native.2d_P1D_mean"]

ICON.native.2d_P1D_mean:
  args:
    chunks: null
    consolidated: true
    urlpath: https://swift.dkrz.de/v1/dkrz_948e7d4bbfbb445fbff5315fc433e36a/easygems_tutorial/ICON.native.2d_P1D_mean.zarr
  description: ''
  driver: intake_xarray.xzarr.ZarrSource
  metadata:
    catalog_dir: https://data.nextgems-h2020.eu

Opening a dataset

Now that we’ve located a dataset, we can load it. Intake supports lazy loading, meaning the data isn’t read until you explicitly request it. The cat.tutorial["ICON.native.2d_P1D_mean"] syntax returns a data source object.

Using the to_dask() method, we can open the dataset:

ds = cat.tutorial["ICON.native.2d_P1D_mean"](chunks={"ncells": -1}).to_dask()
ds

<xarray.Dataset> Size: 844MB
Dimensions:             (time: 31, ncells: 1310720)
Coordinates:
  * time                (time) datetime64[ns] 248B 2020-01-01T23:59:59 ... 20...
    cell_sea_land_mask  (ncells) float64 10MB dask.array<chunksize=(1310720,), meta=np.ndarray>
    lat                 (ncells) float64 10MB dask.array<chunksize=(1310720,), meta=np.ndarray>
    lon                 (ncells) float64 10MB dask.array<chunksize=(1310720,), meta=np.ndarray>
Dimensions without coordinates: ncells
Data variables:
    hus2m               (time, ncells) float32 163MB dask.array<chunksize=(1, 1310720), meta=np.ndarray>
    psl                 (time, ncells) float32 163MB dask.array<chunksize=(1, 1310720), meta=np.ndarray>
    ts                  (time, ncells) float32 163MB dask.array<chunksize=(1, 1310720), meta=np.ndarray>
    uas                 (time, ncells) float32 163MB dask.array<chunksize=(1, 1310720), meta=np.ndarray>
    vas                 (time, ncells) float32 163MB dask.array<chunksize=(1, 1310720), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • time: 31
  • ncells: 1310720
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2020-01-01T23:59:59 ... 2020-01-...

    array(['2020-01-01T23:59:59.000000000', '2020-01-02T23:59:59.000000000',
           '2020-01-03T23:59:59.000000000', '2020-01-04T23:59:59.000000000',
           '2020-01-05T23:59:59.000000000', '2020-01-06T23:59:59.000000000',
           '2020-01-07T23:59:59.000000000', '2020-01-08T23:59:59.000000000',
           '2020-01-09T23:59:59.000000000', '2020-01-10T23:59:59.000000000',
           '2020-01-11T23:59:59.000000000', '2020-01-12T23:59:59.000000000',
           '2020-01-13T23:59:59.000000000', '2020-01-14T23:59:59.000000000',
           '2020-01-15T23:59:59.000000000', '2020-01-16T23:59:59.000000000',
           '2020-01-17T23:59:59.000000000', '2020-01-18T23:59:59.000000000',
           '2020-01-19T23:59:59.000000000', '2020-01-20T23:59:59.000000000',
           '2020-01-21T23:59:59.000000000', '2020-01-22T23:59:59.000000000',
           '2020-01-23T23:59:59.000000000', '2020-01-24T23:59:59.000000000',
           '2020-01-25T23:59:59.000000000', '2020-01-26T23:59:59.000000000',
           '2020-01-27T23:59:59.000000000', '2020-01-28T23:59:59.000000000',
           '2020-01-29T23:59:59.000000000', '2020-01-30T23:59:59.000000000',
           '2020-01-31T23:59:59.000000000'], dtype='datetime64[ns]')

  • cell_sea_land_mask
    (ncells)
    float64
    dask.array<chunksize=(1310720,), meta=np.ndarray>

    long_name :

    sea (-2 inner, -1 boundary) land (2 inner, 1 boundary) mask for the cell

    units :

    2,1,-1,-

    grid_type :

    unstructured

    number_of_grid_in_reference :

    1

    dtype :

    int32

    zlib :

    False

    szip :

    False

    zstd :

    False

    bzip2 :

    False

    blosc :

    False

    shuffle :

    False

    complevel :

    0

    fletcher32 :

    False

    contiguous :

    True

    chunksizes :

    None

    source :

    /work/bm1344/DKRZ/kerchunks_batched/erc2002/icon_grid_0033_R02B08_G.nc

    original_shape :

    [5242880]

    Array Chunk Bytes 10.00 MiB 10.00 MiB Shape (1310720,) (1310720,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • lat
    (ncells)
    float64
    dask.array<chunksize=(1310720,), meta=np.ndarray>

    Array Chunk Bytes 10.00 MiB 10.00 MiB Shape (1310720,) (1310720,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

  • lon
    (ncells)
    float64
    dask.array<chunksize=(1310720,), meta=np.ndarray>

    Array Chunk Bytes 10.00 MiB 10.00 MiB Shape (1310720,) (1310720,) Dask graph 1 chunks in 2 graph layers Data type float64 numpy.ndarray

• Data variables: (5)
  • hus2m
    (time, ncells)
    float32
    dask.array<chunksize=(1, 1310720), meta=np.ndarray>

    CDI_grid_type :

    unstructured

    long_name :

    specific humidity in 2m

    number_of_grid_in_reference :

    1

    param :

    6.0.0

    standard_name :

    qv2m

    units :

    kg kg-1

    Array Chunk Bytes 155.00 MiB 5.00 MiB Shape (31, 1310720) (1, 1310720) Dask graph 31 chunks in 2 graph layers Data type float32 numpy.ndarray

  • psl
    (time, ncells)
    float32
    dask.array<chunksize=(1, 1310720), meta=np.ndarray>

    CDI_grid_type :

    unstructured

    long_name :

    mean sea level pressure

    number_of_grid_in_reference :

    1

    param :

    1.3.0

    standard_name :

    mean sea level pressure

    units :

    Pa

    Array Chunk Bytes 155.00 MiB 5.00 MiB Shape (31, 1310720) (1, 1310720) Dask graph 31 chunks in 2 graph layers Data type float32 numpy.ndarray

  • ts
    (time, ncells)
    float32
    dask.array<chunksize=(1, 1310720), meta=np.ndarray>

    CDI_grid_type :

    unstructured

    long_name :

    surface temperature

    number_of_grid_in_reference :

    1

    param :

    0.0.0

    standard_name :

    surface_temperature

    units :

    K

    Array Chunk Bytes 155.00 MiB 5.00 MiB Shape (31, 1310720) (1, 1310720) Dask graph 31 chunks in 2 graph layers Data type float32 numpy.ndarray

  • uas
    (time, ncells)
    float32
    dask.array<chunksize=(1, 1310720), meta=np.ndarray>

    CDI_grid_type :

    unstructured

    long_name :

    zonal wind in 10m

    number_of_grid_in_reference :

    1

    param :

    2.2.0

    standard_name :

    uas

    units :

    m s-1

    Array Chunk Bytes 155.00 MiB 5.00 MiB Shape (31, 1310720) (1, 1310720) Dask graph 31 chunks in 2 graph layers Data type float32 numpy.ndarray

  • vas
    (time, ncells)
    float32
    dask.array<chunksize=(1, 1310720), meta=np.ndarray>

    CDI_grid_type :

    unstructured

    long_name :

    meridional wind in 10m

    number_of_grid_in_reference :

    1

    param :

    3.2.0

    standard_name :

    vas

    units :

    m s-1

    Array Chunk Bytes 155.00 MiB 5.00 MiB Shape (31, 1310720) (1, 1310720) Dask graph 31 chunks in 2 graph layers Data type float32 numpy.ndarray

What’s happening here?

• chunks={"ncells": -1}: This tells Intake to use Dask for chunking along the ncells dimension.

• .to_dask(): Converts the dataset into a Dask-backed xarray.Dataset, enabling parallel and out-of-core computation.

Dask for memory-intensive applications

Using chunks is crucial for handling large datasets efficiently. Only parts of the data are loaded into memory at a time.

Fetching data for analysis or visualization

Now that the dataset is loaded as a Dask-aware xarray.Dataset, you can perform operations just like with any xarray object. This computes the mean temperature across all cells for a single time step:

ds.ts.isel(time=0).mean("ncells").values

array(286.84082, dtype=float32)

Lazy dataset

Because the dataset is Dask-backed, operations are lazy and won’t load data until the actual computation is triggered.