Continued development of continuum imaging pipelines using RASCIL

 Should be discussed at PI planning to make sure it aligns with the development planned by other teams working on similar pipelines.

Timeboxed activity to continue development and analysis of SDP imaging pipelines based on RASCIL with the aim of informing the development of pipelines for the SKA SDP system.

• As part of this activity, the intention is to develop, make available, and compare pipelines of incrementally increasing complexing to provide a baseline reference for current state-of-the-art imaging that is applicable to SKA and should include (not all expected to be done in PI12 and serval steps may be already complete):

• • Simple DI imaging
  • Imaging with correction for known primary beams
  • Imaging including DDE calibration (due to unknown beam or atmospheric corruption)
  • Imaging with DI calibration prior to the self-cal loop
  • Demonstration of a modest level of parallelisation (eg. up to 5 nodes), possibly packaged as a micro-benchmark in collaboration with the Services ART (ie PLANET)
  • Mapping of the performance bottlenecks in the pipeline against processing functions/processing steps (possibly those identified in SP-1959)

• Needs to adhere to the following unless specifically agreed otherwise:
  • Pipelines developed must be easily discoverable (in SKA GitLab), and runnable by others - including sufficient documentation, installation instructions suitable for another SDP pipeline developer to use without needing unreasonable levels of specific knowledge about the code/application in question.
  • Whenever possible, comparisons should be made with other solutions being developed (ie for SP-1998, SP-1904, SP-1549) and when comparisons are made they should be done using standard tools such as the QA checker. Where these tools are found to be insufficient this ticket should help contribute to improving them and making them available for others to use.
  • Pipeline results should make use of standard SKA simulation data sets (possibly revised in PI12 as part of SP-1901). 
  • Development of the pipelines should be regularly shared as part of the Processing Workflows CoP and major results should continue to be demoed at DP System Demos at the earliest opportunity (possibly not waiting until the end of the PI and the closeout of the ticket if a useful and interesting result is found along the way).

The new simulated continuum imaging data sets generated for SP-1901 have been processed with the RASCIL imager, and the results are described in Continuum Imaging Simulations PI 12: Results.

For the ICAL runs with direction-independent calibration effects, it was necessary to add the capability to use an initial sky model in the analysis.

The large images required for the continuum imaging simulations has led us to investigate of the scaling of the pipeline. Test show that when using the Nifty gridder the processing time scales approximately as the 11/3 power of the image size. This has in led us to to investigate using w-snapshots with the Nifty gridder, although the initial performance appear to be poor.

The large images have also impacted the performance of the imaging QA tool. The poor scaling with image size has led us to parallelise some of the analysis with Dask, and to start investigating alternatives to PyBDSF as a source finder.

A large amount of the RASCIL-related effort has been put into running the workshops to ensure continuity in RASCIL development (see SPO-1322 for details). They have been very fruitful, and we have decided to continue them as a RASCIL developer community of practice.