SDP Initial workflows

This Program Epic begins the collaborative development of SDP workflows with the aim that, by PI14, the DP ART can transition effort towards iteratively developing and improving the performance and scalability of SKA SDP workflows with increasingly large and more realistic data sets obtained from simulations, precursor telescopes, and (eventually) SKA Array Assemblies.

Development in this workstream should demonstrate and provide a test of the core qualities of the SDP architecture; Pipelines and workflows developed should be structured with well-defined levels of abstraction between workflow scripts, execution frameworks, and processing functions. This should result in being able to successfully demonstrate reuse of lower-level processing components with the development of these shared between more than one agile team.

While performance and scaling is not the primary goal, it should not be ignored. By PI14, the expectation is that at least one pipeline or workflow developed will demonstrate minor scaling (~3-10 nodes, 100-300 cores) of at least one relevant stage and there will be at least one pipeline that includes processing functions that make effective use of GPU accelerators.

Solutions developed should be easily discoverable by developers in the SKA software community, must be clearly documented, with sufficient user and developer documentation so that they can be run, tested, and modified easily by other teams than the original code authors. All inputs, parameters and test data used to run workflows should be readily available for testing on a variety of platforms, from developer machines to small compute clusters and CICD infrastructure.

Wherever possible, effort should be made to improve alignment with the wider radio astronomy data processing community outside of SKA, in particular with that of SKA Pathfinders and Precursors.