Details
-
Feature
-
Not Assigned
-
None
-
Data Processing
-
-
2
-
2
-
2
-
Team_SIM
-
Sprint 4
-
-
-
4.5
-
-
DP_epic_1 Team_SIM goal4
Description
Be able to spawn containerised workflows (scripts) programmatically using the k8s helm API.
By taking this approach it should be possible to demonstrate that a workflow script can be executed by the Processing Controller in the exact same way as using the helm CLI.
Once an initial version of the Processing Controller has been developed and tested both independently and as part of the integrated MVP deployment, addition functions such as monitoring and reasoning about resources and queuing of batch workflows could be added to this component. This should first involve a discussion with the architecture team and/or SDP architect(s). Note that also for these considerations there is potentially an overlap with the workflow library and how workflows schedule resources for individual workflow stages.
Proposed steps (not all need to be completed in PI#4)
- Split processing controller from Tango Subarray device
- Dynamic deployment of workflow scripts (Dealing with versions of workflow scripts)
- Move to PI-5: Implement (with real-time workflows) resource negotiation between workflow, processing controller backed by the platform interface API.
- Move to PI-5: Scheduling batch workflow scripts for execution