Loading...

Change Owns to Parent Ofs

Set start and due date...

Xporter

XML

Word

Printable

Details

Type: Feature
Priority: Could have
Fix Version/s: PI14
Component/s: COM SDP SW
Labels:
None

ARTs:

Data Processing
Benefit hypothesis:

Hide

A system for workflow lifecycle management is currently fairly primative before we add more workflows or versions of workflows to SDP a more elaborate solution would be very useful.

Show
A system for workflow lifecycle management is currently fairly primative before we add more workflows or versions of workflows to SDP a more elaborate solution would be very useful.
Acceptance criteria:
Hide

Organise spike with architects and teams involved in control system development (ORCA mostly) to discuss terminology. Document outcomes in Solution Intent, possibly involving re-drawing a more clear version of the workflow management flow diagram.

Enable workflow CI verification in ska-telescope/ska-sdp-daliuge-graphs. This will include a method to define test pass/fail criteria and test methods per workflow. The tests will be driven by a testcafe script, which emulates an actual user loading, translating and submitting a graph to the engine. The verification will use the DAliuGE internal Merkle tree mechanism.

Use the https://gitlab.com/ska-telescope/sdp/ska-sdp-science-pipelines/-/tree/master/src/batch_imaging as an example for how this CI machinery could be extended to automatically deploy workflow (definitions) so they become available to the processing controller in a standard ska-sdp-integration Helm chart deployment.
Show
Organise spike with architects and teams involved in control system development (ORCA mostly) to discuss terminology. Document outcomes in Solution Intent, possibly involving re-drawing a more clear version of the workflow management flow diagram. Enable workflow CI verification in ska-telescope/ska-sdp-daliuge-graphs. This will include a method to define test pass/fail criteria and test methods per workflow. The tests will be driven by a testcafe script, which emulates an actual user loading, translating and submitting a graph to the engine. The verification will use the DAliuGE internal Merkle tree mechanism. Use the https://gitlab.com/ska-telescope/sdp/ska-sdp-science-pipelines/-/tree/master/src/batch_imaging as an example for how this CI machinery could be extended to automatically deploy workflow (definitions) so they become available to the processing controller in a standard ska-sdp-integration Helm chart deployment.
Expect Dependencies:

Inter Program
Feature Points:
5
Initial Size:
5
Cost of Delay:
2
WSJF:
0.4
Epic Link:
SDP AA0.5 system
Agile Teams:

Team_YANDA
Due Sprint:
Sprint 5
Story Point Burn-up:
Overdue:
Outcomes:

Hide

AC1: After a number of meetings, discussions and presentations among the Processing CoP and members of the architecture team, the gist of the name changes are tabled mid-page here. Solution intent has been drafted.

AC2: This initial implemention of a workflow lifecycle management system is based on Merkle trees and graph certification for DALiuGE workflows and was integrated with gitlab CI.

AC3: The DALiuGE Helm chart has been made available (again) in the Helm deployer charts repo and with support from ORCA. The relevant merge requests in the Helm repo (MR22) and sample workflow (MR48).

Show
AC1 : After a number of meetings, discussions and presentations among the Processing CoP and members of the architecture team, the gist of the name changes are tabled mid-page here . Solution intent has been drafted. AC2 : This initial implemention of a workflow lifecycle management system is based on Merkle trees and graph certification for DALiuGE workflows and was integrated with gitlab CI. AC3 : The DALiuGE Helm chart has been made available (again) in the Helm deployer charts repo and with support from ORCA. The relevant merge requests in the Helm repo ( MR22 ) and sample workflow ( MR48 ).
Resolved PI.Sprint:
15.1

Feature Checklist:

Stories Completed, Solution Intent Updated, Outcomes Reviewed, NFRS met, Accepted by FO

Demos:
- DP_ART_15.2
- Processing_CoP
Requirement Status:

PI24 - UNCOVERED
Goals_MIRO:
SPO-1591

Description

Who?

For SDP workflow developers, for AA operators

What?

Improved developer experience for managing SDP workflows and integrating them into SDP
Minimal consideration of workflow provenance information

Why?

As we start adding more workflows or workflow variants to the SDP system we need a well established and easy to use way of adding workflows to the SDP system.

Notes

Improve developer experience for adding and managing SDP workflows
Consideration of minimal workflow provenance data
TBD: Extend to revisiting workflow execution and characterisation with a change of title to "SDP workflow management and execution

---

In particular, reproducibility of data products can be enabled and improved through systematic use of repositories as well as versioning and release functions. Furthermore, collaborative functions allow the gradual build-up of an eco-system of software components. See also https://confluence.skatelescope.org/display/SWSI/DALiuGE+Workflow+Management

Elevating Peter's input from the comment section:
Note that the applicable architectural view here would be the science pipeline management use case view: http://ska-sdp.org/sites/default/files/attachments/ska-tel-sdp-0000013_07_sdparchitecture_usecase_views_0.pdf

I.e. this should make progress on implementing the use case of introducing, validating and deploying a (new) workflow. It is okay if we do this for a particular kind of workflow, however it should iterate from the existing design in https://gitlab.com/ska-telescope/sdp/ska-sdp-science-pipelines with the workflow list https://gitlab.com/ska-telescope/sdp/ska-sdp-science-pipelines/-/blob/master/workflows.json as the main connection point to the SDP LMC.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

ska-tel-sdp-0000013_07-fig2.jpg
157 kB
08/Mar/22 4:28 AM

Issue Links

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

Structure

Activity

People

Assignee:: Wortmann, Peter

Reporter:: Mort, Ben

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Feature Progress

Story Point Burn-up: (100.00%)

Feature Estimate: 5.0

	Issues	Story Points
To Do	0	0.0
In Progress	0	0.0
Complete	17	37.0
Total	17	37.0

Dates

Created:: 15/Feb/22 12:21 AM

Updated:: 15/Apr/24 9:44 AM

Resolved:: 24/Jun/22 3:10 PM

Details

Description

Attachments

Attachments

Issue Links

Structure

Activity

People

Feature Progress

Dates

Structure Helper Panel