Revise release processes for SDP repositories

Making SDP releases takes a lot of effort due to the high number of interdependent artefacts that need to be created. This means that we are at danger of making releases that have internally inconsistent dependencies, which has already caused problems previously. The goal here is to make it easier (ideally automated) to integrate components and create consistent releases.

• release processes for SDP components that will allow us to quickly make collective patch releases of many SDP components to refresh shared dependencies (e.g. configuration library or an external dependency).
  • Ideally covering Helm, OCI image and Python package dependencies
• Adjust CI processes to ensure that such releases are safe (e.g. automatically test with updated dependencies on a reasonable schedule).

Demo planned for PI21

AC1: 

We achieved the following:

• Migrated and renamed the skairt repository to ska-rt: https://gitlab.com/ska-telescope/ska-rt 
• Initial investigation and plan made for the work at the beginning of the PI:
  • https://confluence.skatelescope.org/display/SE/Investigation+into+automating+the+release+process
  • https://confluence.skatelescope.org/display/SE/Proposed+plan+of+process+execution
• Updated ska-rt to:
  • the CLI uses the `–mode` argument to determine what information to use from the skart.toml file for dependency updates. At the moment, this is used to distinguish between release and development version updates: MR4
  • Allow passing in custom configuration to the CLI (overwrites using skart.toml): MR8
  • Various updates and bug-fixes, including, e.g. in the way ska-rt identifies patterns in CI jobs to find release versions: MR1
• Added three new make targets to ska-cicd-makefile. These are located in the dependencies.mk file. We also added relevant tests: MR227, MR228. These are:
  • deps-update: update dependencies with skart (development version mode) and poetry lock
  • deps-update-release: update dependences with skart in release mode and with poetry lock
  • deps-patch-release: updates dependencies -> bumps patch version -> pushes tag (and waits for tag pipeline to finish) -> pushes changes to main
• Added the skart.toml file to every SDP repository listed in ORC-1822 description.

Currently finishing off:

• Recursive behaviour of ska-rt, which allows updating dependencies automatically for a whole dependency tree: MR7
• This should be the final piece of the puzzle to get the automation working.

We will need to address in PI21:

• Need to update documentation in the SDP Integration repository to describe this release process: ORC-1839
• Need to make a first release of ska-rt with the new functionality: ORC-1840

AC2:

• Not achieved. We have not adjusted any of the CI pipelines to use the new release process. This will need to be addressed in PI21.