Contributing to easy.gems#

Great to have you here 👋 and thank you very much for your interest in contributing to easy.gems 👍. We welcome contributions from anyone of any kind, e.g. notes about errors, requests for more articles or better descriptions, questions about specific examples, direct updates to articles or new articles, better illustrations and more.

This document should give you a little guide, how we try to organise these kinds of contributions. Our process of updating easy.gems is closely tied to workflows common to gitlab, so they might be familiar, but probably there’s something new to learn as well. In particular, we are using gitlab.dkrz.de for collaboration. If you don’t already have an DKRZ user account, you’ll have to create one to contribute.

We collect things which are not yet written out in the form of issues. These could be ideas, plans, requests, bug reports etc
 all the things for which you don’t have an immediate solution at hand, want to discuss with someone or seek for others to do it 😁. Thus, if you would like to have some things improved, please have a look at our existing issues, where you might already find similar requests and can add some emphasis or additional input. If you don’t find anything related, please open a new one and describe what you like to see. Each issue opens something like a small chat forum for exchange about the requested topic. It’s also a place to collect required information or further plans if needed.

Changes to the easy.gems website happen via merge requests. A merge request is a bit similar to an issue, as it also opens up a small chat forum. However, it is always tied to a certain set of changes to the source code of easy.gems. As easy.gems technically is a static website, the source code consists of all the things which make up the site, that is the content, the layout and styles and the instructions how to generate the site. Changes to any of these will happen in a git branch (more on that later) and those changes will be reviewed and merged back into the main version of easy.gems using a merge request. Please have a look at our existing merge requests. If you see some open requests on topics you know a bit or two about, please help us out by Reviewing it.

If you want to add your own changes, there are two options: very small changes (e.g. spelling errors) can be made quickly by using the Edit this page button on each page. This will direct you to a little text editor where you can type your changes and which will lead you to the creation of a merge request. For anything larger, you’ll have to get a local copy of the git repository as described in Adding Content.

Adding Content#

New content is added using the git version control system. There are many good resources describing git in general, this document will only cover the basics. If you search for more information, maybe the git book or gitready are good starting points.

The starting point will be the source repository for this documentation at https://gitlab.dkrz.de/easy/gems/. If you haven’t done already, please go there and “request access”.

Once you have access, you can perform small edits on the files directly there (just as if you use the Edit this page button on an article page). For bigger changes, we recommend you clone the repository via SSH:

git clone git@gitlab.dkrz.de:easy/gems.git
cd gems

How can I work with git, SSH-keys and gitlab.dkrz.de?

Once you have logged in on gitlab (same login/password as for levante), you need to set up SSH key auth (follow the advice at https://gitlab.dkrz.de/-/profile/keys for generating ssh keys on your machine). You can register your key on gitlab (same link as above). After that you can run the download command from your local home over terminal.

At this point, you have a fresh local copy of the source files of the easy.gems book on your computer. You can edit this copy as you wish and send these edits back to the main repository afterwards. To do so, you may want to create a new git branch at this point already (git checkout -b, see below), but you can also do this later. For some more specific hints about how to edit pages, please refer to the sections below.

To preview your edits, you need sphinx and some other dependencies:

python3 -m pip install -r requirements.txt

easy.gems environment on Levante

On levante, you can load a python virtual environment:

module load python3
source /work/kv0653/k202134/venvs/easy.gems/bin/activate

To leave it again, call:

deactivate
module rm python3

(You don’t need the python module to start the python from the venv, but it comes with pandoc and possibly other tools required for building the pages).

Further below, you can find some specific hints on adding content in our mainly used formats reStructuredText, MyST Markdown, and Jupyter Notebooks. While we started this project using reStructuredText, we have now opened the possibility of using MyST markdown as this eases moving text between gitlab, pads, and easy.gems. So, both formats work, and for a new page markdown .md probably is the better choice, while for existing pages, the transformation might be a bit tedious. After doing your edits, you are ready to go and can build the book by running:

make html

You’ll find the index page at build/html/index.html, please open it in your favourite web browser. You can repeatedly edit the source files, run make html and refresh the web-page until you are happy with your change. Afterwards, it’s time to send your changes back to the main repository.

For automatic cleanup, we use pre-commit. Please activate it for your local repository once before doing your first commit using:

pre-commit install

If you haven’t done so yet, please create a git branch (you’ll have to choose a name):

git checkout -b MY_CREATIVE_BRANCH_NAME

Afterwards, you’d be adding your changed files to the so called “staging area”, which marks them as to be committed:

git add [MY_CHANGED_FILES]

The next step is to actually commit your changes to your local clone of the repository. For each commit, you should come up with a descriptive message, explaining what you did. If it’s a short one, you can add the message on the command line using the -m option. If you want to explain more, just skip the message and git will open a text editor for you:

git commit -m 'my fancy changes'

Warning

For python code and jupyter notebooks we use the black code formatter to re-format the files. This leads to rejected commits, see below for more info on the workflow.

Now, it’s time to do the actual upload of your branch (you’ll have to specify the same name as above):

git push -u origin MY_CREATIVE_BRANCH_NAME

Finally, please create a merge request for your branch, which will do some checks and build your new version in a “review environment” (after a few minutes, you’ll find a link to “View app” on the merge request page). This will give others the option to have a quick look at what you did, and if everything is fine, your changes will be merged into the current main branch and become publicly visible at https://easy.gems.dkrz.de/.

Adding jupyterlab/ipython notebooks#

You can directly add ipython notebooks to the gitlab repository. They will be parsed by the nbsphinx plugin. The raw notebook can be downloaded via the Page source link at the bottom of the page.

We are using jupytext paired notebooks within Easy Gems, because we want a plain-text notebook format to facilitate reviews. In addition we use the black code formatter to ensure consistently formatted code. These scripts are automatically run by pre-commit if pre-commit and the environment are configured as detailed above.

Usually, your first attempt at committing will be refused, and new changes will be added to your jupyterhub notebook and the matching paired .md plain text version. You will get a message like

(easy.gems)$ git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
(easy.gems)$ git commit -m 'add a duplicate healpix cartopy notebook'

[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /home/XY/.cache/pre-commit/patch123567890.
jupytext.................................................................Failed
- hook id: jupytext
- exit code: 2
- files were modified by this hook

reformatted -

All done!
1 file reformatted.
[jupytext] Reading source/Processing/healpix/healpix_cartopy-Copy1.ipynb in format ipynb
[jupytext] Executing black -
[jupytext] Updating source/Processing/healpix/healpix_cartopy-Copy1.ipynb
[jupytext] Error: the git index is outdated.
Please add the paired notebook with:
git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
[jupytext] Updating source/Processing/healpix/healpix_cartopy-Copy1.md
[jupytext] Error: the git index is outdated.
Please add the paired notebook with:
git add source/Processing/healpix/healpix_cartopy-Copy1.md

black................................................(no files to check)Skipped
[INFO] Restored changes from /home/XY/.cache/pre-commit/patch123567890

From this, take both git add commands, run them, and try to commit again. This time it should work.

git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
git add source/Processing/healpix/healpix_cartopy-Copy1.md
git commit -m 'add a duplicate healpix cartopy notebook'

Adding text articles#

easy.gems uses reStructuredText (reST) or MyST markdown as markup language for text articles. Please refer to the linked documents for introductions to the formats. Using MyST has the advantage that it is mostly compatible with the syntax of gitlab/github and HegdeDoc pads. Below are some further tricks for writing special content blocks in reST:

Adding Math#

In MyST .md documents, you can simply use the LaTeX style $ and $$ syntax. In reST

:math:`\sum_{i=1}^n i^2`

yields \(\sum_{i=1}^n i^2\)

.. math::
   \sum_{i=1}^n i^2

yields

\[ \sum_{i=1}^n i^2 \]

Adding code snippets#

Use the following syntax:

.. code:: python

   print ("hello world")

To produce

print ("hello world")

Note that there needs to be an empty line before the .. code-block:: directive, and the code needs to be indented. Similarly many other languages are supported.

For inline code, you can use

:code:`sample code`

to produce sample code.

Reviewing#

To review a merge request, one would usually go to the individual merge request page (you can check if there’s currently an open one, otherwise please have a look at existing Merged requests). On the Overview page, you’ll find the initial description of the merge request as well as the discussion. On the Changes page, you’ll find a diff-view of the files which actually have been changed. It’s a good idea to look through those changes first, such that you see which parts of the site have been touched and which haven’t. Generally, only those parts which are described on the Overview page should be touched by the changes.

For every commit/merge request, gitlab tries to build a preview version of the easy.gems site with the via the CI/CD pipelines. If this was successful for the merge request at hand, you can find a View app button in the pipeline panel at the top of the overview page, that takes you to the preview version for this merge request, so you can see if everything looks as expected.

When reviewing a merge request, please check if the actual Changes correspond to the description in the Overview and check if the content of the changes looks good to you. You may consider suggestions for material that makes sense to add as part of the merge-request, but only if they are truly germane to the request at hand. If issues which extend beyond the scope of the merge request arise, please open a new issue and reference back to the originating merge request. This will help us to finalize merge requests quickly and not to loose time in overly long discussions.