Details
-
Spike
-
Must have
-
None
-
Data Processing
-
-
-
3
-
3
-
8.333
-
Team_SIM
-
Sprint 3
-
-
-
-
11.6
-
Stories Completed, Integrated, Solution Intent Updated, BDD Testing Passes (no errors), Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
Description
SDP workflows are meant to be easy to add and modify. This requires a certain amount of infrastructure, which is currently (at best) a bit arcane - adding a workflow might require generating and registering two Docker images and adjusting the workflow list, in a mostly manual fashion.
The goal here is to have a spike/workshop with the stakeholders (the developers!) to discuss and identify how this could be improved. Hopefully we will identify some simple measures that can be put into place immediately, larger refactoring work will likely need to happen in later PIs (towards AA0.5 milestone).
Some considerations as starting points:
- How to work with this in a development environment like the famous "developer laptop"?
- What if the laptop isn't enough, and we need to involve a (possibly third-party) HPC centre into testing the workflow? We should be able to do this at SRCs at minimum!
- Especially consider both how to add workflows and execution engines / processing components.
- How would a continuous integration pipeline for workflows look like?
References:
- PI10 I&A Problem 1: https://miro.com/app/board/o9J_lXFWnkE=/