ceph-mirror/doc/dev/continuous-integration.rst

Continuous Integration Architecture
===================================

In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines
are centered around Jenkins. And their configurations are generated using `Jenkins Job Builder`_.

.. _Jenkins Job Builder: https://docs.openstack.org/infra/jenkins-job-builder/

Let's take the ``make check`` performed by Jenkins as an example.

ceph-pull-requests
------------------

``ceph-pull-requests`` is a jenkins job which gets triggered by a GitHub pull
request or a trigger phrase like::

    jenkins test make check

There are multiple parties involved in this jenkins job:

.. graphviz::

   digraph  {
     rankdir="LR";
     github [
      label="<git> git_repo | <webhooks> webhooks | <api> api";
      shape=record;
      href="https://github.com/ceph/ceph";
     ];
     subgraph cluster_lab {
       label="Sepia Lab";
       href="https://wiki.sepia.ceph.com/doku.php";
       shape=circle;
       apt_mirror [
         href="http://apt-mirror.front.sepia.ceph.com";
       ];
       shaman [
         href="https://shaman.ceph.com";
       ];
       chacra [
         peripheries=3;
         href="https://chacra.ceph.com";
       ];
       subgraph cluster_jenkins {
         label="jenkins";
         href="https://jenkins.ceph.com";
         jenkins_controller [ label = "controller" ];
         jenkins_agents [ label = "agents", peripheries=3 ];
       };
     };
     {
       rank=same;
       package_repos [ peripheries=3 ];
       pypi;
       npm;
     }
     github:webhooks -> jenkins_controller [ label = "notify", color = "crimson" ];
     jenkins_controller -> jenkins_agents [ label = "schedule jobs" ];
     jenkins_agents -> github:git [ label = "git pull" ];
     jenkins_agents -> shaman [ label = "query for chacra repo URL" ];
     jenkins_agents -> chacra [ label = "pull build dependencies" ];
     jenkins_agents -> package_repos [ label = "pull build dependencies" ];
     jenkins_agents -> pypi [ label = "pull Python packages" ];
     jenkins_agents -> npm [ label = "pull JavaScript packages" ];
     jenkins_agents -> apt_mirror [ label = "pull build dependencies" ];
     jenkins_agents -> github:api [ label = "update", color = "crimson" ];
   }

Where

Sepia Lab
   `Sepia Lab`_ is a test lab used by the Ceph project. This lab offers
   the storage and computing resources required by our CI infra.

Jenkins agents
   are a set of machines which perform the CI jobs. In this case, they

   #. pull the git repo from GitHub and
   #. rebase the pull request against the latest master
   #. set necessary environment variables
   #. run ``run-make-check.sh``

Chacra
   is a server offering RESTful API allowing the clients to store and
   retrieve binary packages. It also creates the repo for uploaded
   packages automatically. Once a certain repo is created on chacra, the
   configured shaman server is updated as well, then we can query shaman
   for the corresponding repo address. Chacra not only hosts Ceph packages,
   it also hosts quite a few other packages like various build dependencies.

Shaman
   is a server offering RESTful API allowing the clients to query the
   information of repos hosted by chacra nodes. Shaman is also known
   for its `Web UI`_. But please note, shaman does not build the
   packages, it just offers information on the builds.

ceph-perf-pull-requests
-----------------------

``ceph-perf-pull-requests`` runs CBT performance regression checks on dedicated
``performance`` Jenkins agents. The job definition lives in `ceph-build`_ and
generates two jobs from a single template: ``ceph-perf-classic`` and
``ceph-perf-crimson``.

A pull request can trigger a run with::

    jenkins test classic perf
    jenkins test crimson perf

The ``performance`` label is also whitelisted for automatic runs. Each job:

#. checks out ``ceph-main`` (``origin/main``) and the PR merge ref
#. builds both trees (classic ``vstart-base`` or Crimson ``crimson-osd``)
#. runs the checked-in ``radosbench_4K_read.yaml`` workload via ``run-cbt.sh``
#. compares PR results against ``main`` with ``cbt/compare.py``
#. publishes a GitHub check named ``perf-test-{classic,crimson}``

The benchmark YAML is always taken from ``ceph-main`` so PR and baseline runs use
the same workload definition. Teuthology-to-CBT translation is handled by
``src/test/crimson/cbt/t2c.py`` in the Ceph tree (not patched at build time).

.. _ceph-build: https://github.com/ceph/ceph-build

As the following shows, `chacra`_ manages multiple projects whose metadata
are stored in a database. These metadata are exposed via Shaman as a web
service. `chacractl`_ is a utility to interact with the `chacra`_ service.

.. graphviz::

   digraph  {
     libboost [
       shape=cylinder;
     ];
     libzbd [
       shape=cylinder;
     ];
     other_repos [
       label="...";
       shape=cylinder;
     ];
     postgresql [
       shape=cylinder;
       style=filled;
     ]
     shaman -> postgresql;
     chacra -> postgresql;
     chacractl -> chacra;
     chacra -> libboost;
     chacra -> libzbd;
     chacra -> other_repos;
   }

.. _Sepia Lab: https://wiki.sepia.ceph.com/doku.php
.. _Web UI: https://shaman.ceph.com

build dependencies
------------------

Just like lots of other software projects, Ceph has both build-time and
run-time dependencies. Most of time, we are inclined to use the packages
prebuilt by the distro. But there are cases where

- the necessary dependencies are either missing in the distro, or
- their versions are too old, or
- they are packaged without some important feature enabled.
- we want to ensure that the version of a certain runtime dependency is
  identical to the one we tested in our lab.

No matter what the reason is, we either need to build them from source, or
to package them as binary packages instead of using the ones shipped by the
distro. Quite a few build-time dependencies are included as git submodules,
but in order to avoid rebuilding these dependencies repeatedly, we pre-built
some of them and uploaded them to our own repos. So, when performing
``make check``, the building hosts in our CI just pull them from our internal
repos hosting these packages instead of building them.

So far, following packages are prebuilt for ubuntu focal, and then uploaded to
`chacra`_:

libboost
    packages `boost`_. The packages' names are changed from ``libboost-*`` to
    ``ceph-libboost-*``, and they are instead installed into ``/opt/ceph``, so
    they don't interfere with the official ``libboost`` packages shipped by
    distro. Its build scripts are hosted at https://github.com/ceph/ceph-boost.
    See https://github.com/ceph/ceph-boost/commit/2a8ae02932b2a1fd6a68072da8ca0df2b99b805c
    for an example of how to bump the version number. The commands used to
    build 1.79 on a vanilla Ubuntu Focal OS are below.

    .. prompt:: bash $

       sudo apt install debhelper dctrl-tools chrpath libbz2-dev libicu-dev bison \
         flex docbook-to-man help2man xsltproc doxygen dh-python python3-all-dev graphviz
       wget http://download.ceph.com/qa/boost_1_79_0.tar.bz2
       git clone https://github.com/ceph/ceph-boost
       tar xjf boost_1_79_0.tar.bz2
       cp -ra ceph-boost/debian boost_1_79_0/
       pushd boost_1_79_0
       export DEB_BUILD_OPTIONS='parallel=6 nodoc'
       dpkg-buildpackage -us -uc -b
       popd
       BOOST_SHA=$(git ls-remote https://github.com/ceph/ceph-boost main | awk '{ print $1 }')
       ls *.deb | chacractl binary create \
         libboost/master/$BOOST_SHA/ubuntu/focal/amd64/flavors/default

libzbd
    packages `libzbd`_ . The upstream libzbd includes debian packaging already.

libpmem
    packages `pmdk`_ .

.. note::

   please ensure that the package version and the release number of the
   packaging are properly updated when updating/upgrading the packaging,
   otherwise it would be difficult to tell which version of the package
   is installed. We check the package version before trying to upgrade
   it in ``install-deps.sh``.

.. _boost: https://www.boost.org
.. _libzbd: https://github.com/westerndigitalcorporation/libzbd
.. _pmdk: https://github.com/pmem/pmdk

But in addition to these libraries, ``ceph-mgr-dashboard``'s frontend uses lots of
JavaScript packages. Quite a few of them are not packaged by distros. Not to
mention the trouble of testing different combination of versions of these
packages. So we decided to include these JavaScript packages in our dist tarball
using ``make-dist``.

Also, because our downstream might not want to use the prepackaged binaries when
redistributing the precompiled Ceph packages, we also need to include these
libraries in our dist tarball. They are

- boost
- liburing
- pmdk

``make-dist`` is a script used by our CI pipeline to create dist tarball so the
tarball can be used to build the Ceph packages in a clean room environment. When
we need to upgrade these third party libraries, we should

- update the CMake script
- rebuild the prebuilt packages and
- update this script to reflect the change.

Uploading Dependencies
----------------------

To ensure that prebuilt packages are available by the jenkins agents, we need to
upload them to either ``apt-mirror.front.sepia.ceph.com`` or `chacra`_. To upload
packages to the former would require the help of our lab administrator, so if we
want to maintain the package repositories on regular basis, a better choice would be
to manage them using `chacractl`_. `chacra`_ represents packages repositories using
a resource hierarchy, like::

  <project>/<branch>/<ref>/<distro>/<distro-version>/<arch>

In which:

project
    in general, it is used for denoting a set of related packages. For instance,
    ``libboost``.

branch
    branch of project. This mirrors the concept of a Git repo.

ref
    a unique id of a given version of a set packages. This id is used to reference
    the set packages under the ``<project>/<branch>``. It is a good practice to
    version the packaging recipes, like the ``debian`` directory for building DEB
    packages and the ``spec`` for building RPM packages, and use the SHA1 of the
    packaging recipe for the ``ref``. But you could also use a random string for
    ``ref``, like the tag name of the built source tree.

distro
    the distro name for which the packages are built. Currently, following distros are
    supported:

    - centos
    - debian
    - fedora
    - rhel
    - ubuntu

distro-version
    the version of the distro. For instance, if a package is built on ubuntu focal,
    the ``distro-version`` should be ``20.04``.

arch
    the architecture of the packages. It could be:

    - arm64
    - amd64
    - noarch

So, for example, we can upload the prebuilt boost packages to chacra like

.. prompt:: bash $

   ls *.deb | chacractl binary create \
     libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default

.. _chacra: https://github.com/ceph/chacra
.. _chacractl: https://github.com/ceph/chacractl

Update ``install-deps.sh``
--------------------------

We also need to update ``install-deps.sh`` to point the built script to the new
repo. Please refer to the `script <https://github.com/ceph/ceph/blob/master/install-deps.sh>`_,
for more details.