.. _manage-task-configuration:

=========================
Manage task configuration
=========================

See :ref:`explanation-task-configuration` for the basic concepts.

Introduction
============

More concretely when applying task configuration, Debusine will retrieve
a sorted list of :bare-data:`debusine:task-configuration` entries matching
the work request to configure and apply the instructions contained there.

The matching criteria are:

* the task type (required)
* the task name (required)
* the "subject" of the task
* the "configuration context" of the task
* the scheduler tags

.. note::

    Subject, configuration context and scheduler tags are data that are
    dynamically computed by each task. You can review the generated data
    in the "Internals" tab of each work request. While the scheduler tags
    have their own dedicated section, the subject and context are part of
    "Dynamic data" section and can be found under the "subject" and
    "configuration_context" field respectively (beware to not mix
    with the "runtime_context" which is used for managing task history).

The possible operations are:

* setting a default value to a field
* overriding the value of a field
* deleting the value of a field
* locking the value of a field (forbidding further overrides)
* requiring additional scheduler tags


Editing through text files
==========================

Debusine allows maintaining the :collection:`debusine:task-configuration`
collection in text files in a local directory, preferably a ``git``
repository.

To initialize a task configuration git repository, follow these steps.
If you wish, you can follow the same process without using git, just
creating an empty directory to work in, and skipping the git steps.

#. Create a new git repository:

   .. code-block:: console

    $ git init task-config
    $ cd task-config

#. Pull the current collection contents into the repository:

   .. code-block:: console

    $ debusine --server NAME task-config pull --workspace WORKSPACE
    0 added, 0 updated, 0 deleted, 0 unchanged

#. This will create a ``MANIFEST`` file, and YAML files for any existing
   configuration.
   The workspace name is saved in this file, so we won't need to specify
   it again, when working with this repository.
#. Commit all of this to git:

   .. code-block:: console

    $ git add .
    $ git commit -m "Initialize workspace configuration repository"

#. Add configuration by making new YAML files, or editing existing files.
#. Commit the changes.
#. Push them back to the configuration collection:

   .. code-block:: console

    $ debusine --server NAME task-config push
    Pushing data to server...
    0 added, 0 updated, 0 removed, 1 unchanged

#. Push your git repo to somewhere that's shared with the other
   workspace admins.

The client will protect you from overwriting configuration if the
current server configuration is not in your git repository's history:

.. code-block:: console

    $ debusine --server NAME task-config push
    server collection was pushed from commit 148a8269de622667b7635d63c505b15a9ab8381d which is not known to ~/task-config

If you need to override this (maybe because you rewrote git history),
you can:

.. code-block:: console

    $ debusine --server NAME task-config push --force
    server collection was pushed from commit 148a8269de622667b7635d63c505b15a9ab8381d which is not known to ~/task-config
    Pushing data to server...
    0 added, 0 updated, 0 removed, 1 unchanged


Naming of files
---------------

``debusine:task-configuration`` entries pulled from the server are stored as
YAML files. The file names are ignored when matching remote entries with local
ones, so you can rename files and organize them into directories as you find it
convenient.

Multiple rules can be contained in a single file, by making a YAML list
of rules in one file.

When new entries are found on the server during a pull, they are stored inside
a ``new/`` directory from which they can be safely dispatched to a more
meaningful location in the checkout.

When pulling changes on existing files, debusine is able to locate the local
files according to their contents and update them in place.


Git and non-git
---------------

If the local checkout is in git:

* Push and pull refuse to work if there are uncommitted changes to ``.yaml``
  files.
* If an entry has been deleted on the server, it will also be deleted locally.
* Push will refuse to work if the server contents were pushed from git, and
  their commit hash is not an ancestor of the local git commit

If the local checkout is not in git:

* If an entry has been deleted on the server, it will not be deleted locally to
  prevent accidental data loss, although it will be reported in a warning.

You can use ``--force`` to bypass the checks above.


Dry run
-------

If using ``--dry-run``, push and pull operations will go through all the
motions, including download and upload of data and reporting of change
statistics, but changes are not stored in the database.


.. _example-task-configuration:

Example task configuration files
================================

YAML files in the checkout are the :bare-data:`debusine:task-configuration`
serialized to YAML (or list of those).

Here are a few examples of task configuration items. You can find additional
real-word examples in the `debusine.debian.net configuration repository
<https://salsa.debian.org/freexian-team/debusine.debian.net/workspace-developers>`__.

For simplicity, all the examples below have all the content in a single
file, but you can split the various entries across multiple files as you
see fit, in particular to better control the order in which they are
processed (see :ref:`task-configuration-ordering`) since when there's
overlap, the order is particularly important.

Example 1: overriding a value for a field
-----------------------------------------

Here's how you would override sbuild's ``build_options`` to ``parallel=2``
when it is building the ``openjdk-17`` source package. Note that if the
parent workflow starting the task is defining a value for the ``build_options``
field, that value would not be used, it is replaced by the value provided
through this mechanism.

.. code-block:: yaml

    task_type: Worker
    task_name: sbuild
    tags_provided:
      - task:source-package:openjdk-17
    override_values:
      build_options: parallel=2
    comment: reduce parallelism to avoid crashing workers

Note that we rely on the scheduler tags here to filter the source package,
but we could have use the subject field too (``subject: openjdk-17``) since
many tasks use the source package as subject.

You can add additional tags in ``tags_provided`` to restrict even further the
work requests where it would be applied. Debusine generate multiple tags to let
you match work requests by scope, workspace, user, groups of the user, etc.
See :ref:`scheduler-tags-task`.

Example 2: setting a default value for a field
----------------------------------------------

With the entries below, ``lintian`` tasks will be provided
a default value for the ``exclude_tags`` field except
when it is run in the "sid" distribution (where ``delete_values``
effectively cancels the former entry).

But if the user starting the workflow provided a value of his own,
or if the workflow template already provided a default value, then
that value would prevail over the value provided through the task
configuration mechanism.

.. code-block:: yaml

    - task_type: Worker
      task_name: lintian
      default_values:
        exclude_tags:
          - source-nmu-has-incorrect-version-number
          - older-source-format
      comment: By default for stable releases, ignore some tags.

    - task_type: Worker
      task_name: lintian
      context: sid
      delete_values:
        - exclude_tags
      comment: For sid/unstable, we don't want to ignore tags.

Example 3: locking a value when overlap is expected
---------------------------------------------------

Imagine that ``openjdk-17`` needs a custom ``build_profiles`` value
and you want to ensure that further task configuration (for example
one applying to all packages on a given architecture) will not overwrite
that value. You can do that by using ``lock_values`` in addition the
``default_values`` or ``override_values``.

.. code-block:: yaml

    - task_type: Worker
      task_name: sbuild
      tags_provided:
        - task:source-package:openjdk-17
      override_values:
        build_profiles:
          - nodoc
      lock_values:
        - build_profiles
      comment: we really need nodoc profile or it fails to build see bug 312

    - task_type: Worker
      task_name: sbuild
      tags_required:
        - worker:build-arch:riscv64
      override_values:
        build_profiles:
          - nocheck
      comment: Skip checks while we're bootstrapping the architecture

Note that those two task configuration entries would likely be stored
in two different files in practice. The net result is that any build
of openjdk-17 will always use the "nodoc" build profile, and any
riscv64 build (except for openjdk-17) will use the "nocheck" build profile.

Without ``lock_values``, a risc64 build of openjdk-17 would also use the
"nocheck" build profile because the latter entry would override the change
made by the first entry.

Example 4: use templates to share configuration
-----------------------------------------------

In many cases, we need to apply the same configuration in multiple contexts
(source packages, target distributions, etc.). Template task configuration
entries serve that purpose. The example below configures multiple packages
to go through a signing step in support of secure boot.

Let's create the templates in a first file:

.. code-block:: yaml

    - template: uefi-sign
      default_values:
        enable_make_signed_source: True
        make_signed_source_purpose: uefi

    - template: uefi-sign-with-fwupd-key
      use_templates:
        - uefi-sign
      default_values:
        make_signed_source_key: AEC1234

    - template: uefi-sign-with-grub-key
      use_templates:
        - uefi-sign
      default_values:
        make_signed_source_key: CBD3214

In a second file, let's use those templates to configure the ``debian_pipeline``
workflow of specific packages:

.. code-block:: yaml

    - task_type: Workflow
      task_name: debian-pipeline
      subject: fwupd-efi
      use_templates:
        - sign-with-fwupd-key

    - task_type: Workflow
      task_name: debian-pipeline
      subject: fwupdate
      use_templates:
        - sign-with-fwupd-key

    - task_type: Workflow
      task_name: debian-pipeline
      subject: grub2
      use_templates:
        - sign-with-grub-key

.. note::

    The ``debian_pipeline`` workflow currently doesn't provide
    the ``task:source-package:SOURCE`` scheduler tag and we have
    to rely on matching via the ``subject`` attribute.

Example 5: require an additional tag from the worker
----------------------------------------------------

The example below introduces the ``require_tags`` field which lets
you specify additional :ref:`scheduler tags that the worker must
have <scheduler-tags-worker>`. Let's build a list of source packages
where sbuild and autopkgtest tasks will run on workers that have been
marked by the administrator as being "large".

.. code-block:: yaml

    - template: large-worker
      require_tags:
        - worker:class:large

    - task_type: Worker
      task_name: sbuild
      subject: firefox
      use_templates:
        - large-worker

    - task_type: Worker
      task_name: autopkgtest
      subject: firefox
      use_templates:
        - large-worker

    [ ... repeat for other source packages ... ]

.. note::

    The Debusine tooling doesn't support any templating language that would
    make it easier to create a large number of similar entries. But you can
    build around it since only ``*.yaml`` files are processed, so you could have
    ``*.yaml.in`` templates and process them with your favorite templating language
    as a first step before running ``debusine task-config push``.

..
  Quick way to test these entries on ``debusine-admin shell``:

     from debusine.artifact.models import DebusineTaskConfiguration
     import yaml

     DebusineTaskConfiguration(**yaml.safe_load("""
     …entry…
     """))
