Details
-
Feature
-
Must have
-
None
-
Data Processing
-
-
-
Intra Program
-
13
-
13
-
0
-
Team_HIPPO
-
Sprint 5
-
-
-
-
19.5
-
Outcomes Reviewed
Description
Self-calibration and imaging pipeline for MID demonstrating the scaling to overall performance necessary AA2+ and which we can iterate to become the final ICAl for Mid. Should support direction-dependent calibration or pointing self cal.
Who?
- SDP Software stakeholders
What? (outcomes)
- A pipeline that we can iterate toward demonstration of scaling necessary for AA2
- Working scripts with input data sets to demonstrate additional functionality
- As a starting point this pipeline could perhaps be constructed by evolving the LOFAR software stack, but this is subject to a feasibility assessment
- Descriptions of how (and where) to run the pipeline
- The strategy for scaling is according to SDP Software architecture, e.g., using Dask.
Why?
- The key challenge of SKA data processing that we have identified in advance is the scaling challenge. Even at AA2 observations are going to be of order 100 TBs, which is difficult to process in good time with existing software. Demonstrating the scaling to AA2 scale in test by Q1 2023 is essential in order to have confidence of scaling to the full SKA scale, in production, in time for the completion of the arrays.
- Self-calibration and imaging are core SDP science workflow functions and to stay on schedule to deliver this functionality meeting the performance and scaling requirements for later AAs and full array operation we believe we need to reach ~AA2 scale
References
- PI15 workflow:
- SDP workflows (SI): https://confluence.skatelescope.org/pages/viewpage.action?pageId=161359526
- SDP Science Pipeline Workflow View
