Contributing to easy.gems#
Great to have you here đ and thank you very much for your interest in contributing to easy.gems đ. We welcome contributions from anyone of any kind, e.g. notes about errors, requests for more articles or better descriptions, questions about specific examples, direct updates to articles or new articles, better illustrations and more.
This document should give you a little guide, how we try to organise these kinds of contributions. Our process of updating easy.gems is closely tied to workflows common to gitlab, so they might be familiar, but probably thereâs something new to learn as well. In particular, we are using gitlab.dkrz.de for collaboration. If you donât already have an DKRZ user account, youâll have to create one to contribute.
We collect things which are not yet written out in the form of issues. These could be ideas, plans, requests, bug reports etc⊠all the things for which you donât have an immediate solution at hand, want to discuss with someone or seek for others to do it đ. Thus, if you would like to have some things improved, please have a look at our existing issues, where you might already find similar requests and can add some emphasis or additional input. If you donât find anything related, please open a new one and describe what you like to see. Each issue opens something like a small chat forum for exchange about the requested topic. Itâs also a place to collect required information or further plans if needed.
Changes to the easy.gems website happen via merge requests. A merge request is a bit similar to an issue, as it also opens up a small chat forum.
However, it is always tied to a certain set of changes to the source code of easy.gems. As easy.gems technically is a static website, the source code consists of all the things which make up the site, that is the content, the layout and styles and the instructions how to generate the site. Changes to any of these will happen in a git branch
(more on that later) and those changes will be reviewed and merged back into the main
version of easy.gems using a merge request.
Please have a look at our existing merge requests. If you see some open requests on topics you know a bit or two about, please help us out by Reviewing it.
If you want to add your own changes, there are two options: very small changes (e.g. spelling errors) can be made quickly by using the Edit this page button on each page.
This will direct you to a little text editor where you can type your changes and which will lead you to the creation of a merge request.
For anything larger, youâll have to get a local copy of the git
repository as described in Adding Content.
Adding Content#
New content is added using the git version control system. There are many good resources describing git in general, this document will only cover the basics. If you search for more information, maybe the git book or gitready are good starting points.
The starting point will be the source repository for this documentation at https://gitlab.dkrz.de/easy/gems/. If you havenât done already, please go there and ârequest accessâ.
Once you have access, you can perform small edits on the files directly there (just as if you use the Edit this page button on an article page). For bigger changes, we recommend you clone the repository via SSH:
git clone git@gitlab.dkrz.de:easy/gems.git
cd gems
How can I work with git, SSH-keys and gitlab.dkrz.de?
Once you have logged in on gitlab (same login/password as for levante), you need to set up SSH key auth (follow the advice at https://gitlab.dkrz.de/-/profile/keys for generating ssh keys on your machine). You can register your key on gitlab (same link as above). After that you can run the download command from your local home over terminal.
At this point, you have a fresh local copy of the source files of the easy.gems book on your computer.
You can edit this copy as you wish and send these edits back to the main repository afterwards.
To do so, you may want to create a new git branch at this point already (git checkout -b
, see below),
but you can also do this later.
For some more specific hints about how to edit pages, please refer to the sections below.
To preview your edits, you need sphinx and some other dependencies:
python3 -m pip install -r requirements.txt
easy.gems environment on Levante
On levante, you can load a python virtual environment:
module load python3
source /work/kv0653/k202134/venvs/easy.gems/bin/activate
To leave it again, call:
deactivate
module rm python3
(You donât need the python module to start the python from the venv, but it comes with pandoc and possibly other tools required for building the pages).
Further below, you can find some specific hints on adding content in our mainly used formats
reStructuredText, MyST Markdown, and Jupyter Notebooks. While we started this project using reStructuredText, we have now opened the possibility of using MyST markdown as this eases moving text between gitlab, pads, and easy.gems. So, both formats work, and for a new page markdown .md
probably is the better choice, while for existing pages, the transformation might be a bit tedious.
After doing your edits, you are ready to go and can build the book by running:
make html
Youâll find the index page at build/html/index.html
, please open it in your favourite web browser.
You can repeatedly edit the source files, run make html
and refresh the web-page until you are happy
with your change. Afterwards, itâs time to send your changes back to the main repository.
For automatic cleanup, we use pre-commit. Please activate it for your local repository once before doing your first commit using:
pre-commit install
If you havenât done so yet, please create a git branch (youâll have to choose a name):
git checkout -b MY_CREATIVE_BRANCH_NAME
Afterwards, youâd be adding your changed files to the so called âstaging areaâ, which marks them as to be committed:
git add [MY_CHANGED_FILES]
The next step is to actually commit your changes to your local clone of the repository. For each commit,
you should come up with a descriptive message, explaining what you did. If itâs a short one, you can add
the message on the command line using the -m
option. If you want to explain more, just skip the message
and git
will open a text editor for you:
git commit -m 'my fancy changes'
Warning
For python code and jupyter notebooks we use the black code formatter to re-format the files. This leads to rejected commits, see below for more info on the workflow.
Now, itâs time to do the actual upload of your branch (youâll have to specify the same name as above):
git push -u origin MY_CREATIVE_BRANCH_NAME
Finally, please create a merge request for your branch,
which will do some checks and build your new version in a âreview environmentâ (after a few minutes, youâll
find a link to âView appâ on the merge request page). This will give others the option to have a quick look at
what you did, and if everything is fine, your changes will be merged into the current main
branch and become
publicly visible at https://easy.gems.dkrz.de/.
Adding jupyterlab/ipython notebooks#
You can directly add ipython notebooks to the gitlab repository. They will be parsed by the nbsphinx plugin. The raw notebook can be downloaded via the Page source link at the bottom of the page.
We are using jupytext paired notebooks within Easy Gems, because we want a plain-text notebook format to facilitate reviews. In addition we use the black code formatter to ensure consistently formatted code. These scripts are automatically run by pre-commit if pre-commit and the environment are configured as detailed above.
Usually, your first attempt at committing will be refused, and new changes will be added to your jupyterhub notebook and the matching paired .md
plain text version. You will get a message like
(easy.gems)$ git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
(easy.gems)$ git commit -m 'add a duplicate healpix cartopy notebook'
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to /home/XY/.cache/pre-commit/patch123567890.
jupytext.................................................................Failed
- hook id: jupytext
- exit code: 2
- files were modified by this hook
reformatted -
All done!
1 file reformatted.
[jupytext] Reading source/Processing/healpix/healpix_cartopy-Copy1.ipynb in format ipynb
[jupytext] Executing black -
[jupytext] Updating source/Processing/healpix/healpix_cartopy-Copy1.ipynb
[jupytext] Error: the git index is outdated.
Please add the paired notebook with:
git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
[jupytext] Updating source/Processing/healpix/healpix_cartopy-Copy1.md
[jupytext] Error: the git index is outdated.
Please add the paired notebook with:
git add source/Processing/healpix/healpix_cartopy-Copy1.md
black................................................(no files to check)Skipped
[INFO] Restored changes from /home/XY/.cache/pre-commit/patch123567890
From this, take both git add commands, run them, and try to commit again. This time it should work.
git add source/Processing/healpix/healpix_cartopy-Copy1.ipynb
git add source/Processing/healpix/healpix_cartopy-Copy1.md
git commit -m 'add a duplicate healpix cartopy notebook'
Links between notebooks#
If you want to link between notebooks within easy.gems, please add relative links in the Markdown parts of a notebook like:
[name of the link](../relative/path_to/notebook.ipynb)
This way, links will also work if notebooks are run interactively in jupyterhub. They will be converted appropriately for the rendered easy.gems website and they will also do the right thing for review builds.
Adding text articles#
easy.gems uses reStructuredText (reST) or MyST markdown as markup language for text articles. Please refer to the linked documents for introductions to the formats. Using MyST has the advantage that it is mostly compatible with the syntax of gitlab/github and HegdeDoc pads. Below are some further tricks for writing special content blocks in reST:
Adding Math#
In MyST .md
documents, you can simply use the LaTeX style $
and $$
syntax. In reST
:math:`\sum_{i=1}^n i^2`
yields \(\sum_{i=1}^n i^2\)
.. math::
\sum_{i=1}^n i^2
yields
Adding code snippets#
Use the following syntax:
.. code:: python
print ("hello world")
To produce
print ("hello world")
Note that there needs to be an empty line before the .. code-block::
directive, and the code needs to be indented. Similarly many other languages are supported.
For inline code, you can use
:code:`sample code`
to produce sample code
.
Reviewing#
To review a merge request, one would usually go to the individual merge request page (you can check if thereâs currently an open one, otherwise please have a look at existing Merged requests). On the Overview page, youâll find the initial description of the merge request as well as the discussion. On the Changes page, youâll find a diff-view of the files which actually have been changed. Itâs a good idea to look through those changes first, such that you see which parts of the site have been touched and which havenât. Generally, only those parts which are described on the Overview page should be touched by the changes.
For every commit/merge request, gitlab tries to build a preview version of the easy.gems site with the via the CI/CD pipelines. If this was successful for the merge request at hand, you can find a View app button in the pipeline panel at the top of the overview page, that takes you to the preview version for this merge request, so you can see if everything looks as expected.
When reviewing a merge request, please check if the actual Changes correspond to the description in the Overview and check if the content of the changes looks good to you. You may consider suggestions for material that makes sense to add as part of the merge-request, but only if they are truly germane to the request at hand. If issues which extend beyond the scope of the merge request arise, please open a new issue and reference back to the originating merge request. This will help us to finalize merge requests quickly and not to loose time in overly long discussions.
Useful Links#
Sphinx tutorial
reST & Sphinx cheat sheet