Show
Summary:
We don't believe all AC's (and associated objective) has been met as currently there isn't a released pipeline that can do both gridding and de-gridding in a distributed way on AA2+ scale datasets. Summary of PI23 work:
Refactored existing functionality in notebook form into working scripts including tests, documentation and tutorials.
Created a release (v0.1.0) Releases · SKAO / Science Data Processor / Science Pipeline Workflows / ska-sdp-distributed-self-cal-prototype · GitLab and docs are here SKA SDP Distributed Self-Cal Prototype — Distributed Self-Cal Prototype documentation (skao.int) . User can now install pipeline and perform dirty imaging in a distributed fashion.
De-gridding and subtract can be performed (pan-219 branch) on the CLI using FITS model image and visibilities as input, but not in a distributed way.
Created new release of Swiftly, solving dependency issues between Swiftly and PFL
Made optimisations to XRADIO to allow conversion of AA2 scale datasets https://confluence.skatelescope.org/display/SE/2024-07-31+DP+ART+System+Demo+23.4
See main branch (predict and subtract not yet merged) Files · main · SKAO / Science Data Processor / Science Pipeline Workflows / ska-sdp-distributed-self-cal-prototype · GitLab
Acceptance Criteria (outcomes in green )
Implement imaging portion of a major loop (predict + invert, so from model images to dirty image) using visibility streaming while holding facet data in-memory in a distributed fashion. Made progress. Visibilities -> dirty image is implemented. Predict functionality implemented but not fully integrated, and not tested in a distributed fashion.
Integration of processing functions with storage interface and workflow Integrated ska-sdp-func for de-gridding. Exists in branch notebooks/degrid_wtower_test.py · pan-219-predict-visibilities · SKAO / Science Data Processor / Science Pipeline Workflows / ska-sdp-distributed-self-cal-prototype · GitLab
Address wide-field issues (w-stacking)
Demonstrate on AA2+-scale datasets Not achieved because of bugs and schema changes to MSv4 and broken build systems on ska-sdp-func. New changes to xradio (which can convert large MSv2, but includes schema-breaking changes) means significant refactoring of existing code.
Investigate and optimise performance
Considering scheduler, network and storage throughput Not done as we are awaiting pipeline functionality and ability to convert AA2+ datasets for proper metrics.
Investigate hybridisation of pipeline (GPU processing functions).