Details
-
Feature
-
Should have
-
None
-
Data Processing
-
- Extending our PSS CI coverage to include running executables on specialised hardware
- Triggering CI pipelines directly from panda and cheetah repository events (e.g commit/MR/Merge)
-
-
2.5
-
2.5
-
0
-
Team_PSS
-
Sprint 5
-
-
-
We fully setup our Quad machine to act as a gitlab runner for non Cuda and Cuda builds, in release and debug modes for Panda and Cheetah and are now in the situation where these run on feature branches and the dev branch whenever there are commits.
The runners publish the currently relevant tags to be picked up by our CICD pipelines.
We had a couple of issues with testing in both cases. There is an issue with the drivers on the Quad machine so on this machine the Panda Cuda tests fail for this reason only the non Cuda tests work fine. The Cheetah unit tests we set up to run using the gitlab runner on Quad but due to some of them failing intermittently we decided to remove from the configuration for the time being.
We used a combination of manual installs (e.g. Cuda, boost) and Ansible recipes (gitlab runner) in the end to achieve the configuration of Quad for now. A number of the recipes required for future (e.g. boost, cmake) were written but haven't yet made it passed MR into the dev branch however the gitlab runner recipe is fully useable.
MR's for Panda and Cheetah:
https://gitlab.com/ska-telescope/pss/ska-pss-panda/-/merge_requests/1
https://gitlab.com/ska-telescope/pss/ska-pss-cheetah/-/merge_requests/24
Screen shots of Panda and Cheetah pipelines respectively:
We fully setup our Quad machine to act as a gitlab runner for non Cuda and Cuda builds, in release and debug modes for Panda and Cheetah and are now in the situation where these run on feature branches and the dev branch whenever there are commits. The runners publish the currently relevant tags to be picked up by our CICD pipelines. We had a couple of issues with testing in both cases. There is an issue with the drivers on the Quad machine so on this machine the Panda Cuda tests fail for this reason only the non Cuda tests work fine. The Cheetah unit tests we set up to run using the gitlab runner on Quad but due to some of them failing intermittently we decided to remove from the configuration for the time being. We used a combination of manual installs (e.g. Cuda, boost) and Ansible recipes (gitlab runner) in the end to achieve the configuration of Quad for now. A number of the recipes required for future (e.g. boost, cmake) were written but haven't yet made it passed MR into the dev branch however the gitlab runner recipe is fully useable. MR's for Panda and Cheetah: https://gitlab.com/ska-telescope/pss/ska-pss-panda/-/merge_requests/1 https://gitlab.com/ska-telescope/pss/ska-pss-cheetah/-/merge_requests/24 Screen shots of Panda and Cheetah pipelines respectively: -
18.1
-
Stories Completed, Outcomes Reviewed, Satisfies Acceptance Criteria, Accepted by FO
Description
- Create verified and tested ansible recipes to configure gitlab CI runners suitable for running the PSS software product (within the ska-pss-ci-systems repo) These recipes will be deployed on Kelvin and Tengu.
- Extend existing PSS gitlab runners kelvin and tengu to be compliant with these recipies thus enabling them to publish the full spectrum of avaialble resources as gitlab tags (currently they only advertise cuda).
- Write Gitlab manifest for building branches of panda and cheetah for a single spin tailored to each of our reference platforms. The spin chosen will depend on the available resources advertised by the runners and we can scale up manually as ansible recipies are added. There will need to be a trigger for executing this manifest. Ideally this would be on commit, but given the nature of the runners as developer machines we may resort to overnight builds.
- [STRETCH] A manual trigger with the option to specify the explcit version of the panda and cheetah branches will also be made avaialble. Note that this manifest is a stop-gap solution pending completion of a full coverage & cloud efficient solution being built in the pss-pipeline repository
- [STRETCH] Setup triggers in the cheetah repository to execute the exisiting pss-pipeline repository ci pipeline whenever the above manifest is run.
We can't rely on the using SKAO gitlab cloud based docker images for the Cuda or FPGA executable parts of our CICD so we need a reference platform setup for our specific CICD needs. We can then configure runners to run on this platform.
