Continue development of SKA-Low self-calibration pipeline at AA2+ scale

A configurable and maintainable self-calibration workflow for doing direction-dependent self-calibration and imaging at AA2+ scale is needed for SKA-LOW to demonstrate the SDP strategy for scaling beyond current data processing systems and to SKA scale. 

The current epic completes well before AA2 starts observing; the goal is not complete functionality but rather a demonstration of the scaling on the challenging problem.

• A workflow is developed that performs direction-independent calibration, direction-dependent calibration, imaging and deconvolution.
• Multiple predicts of the same model visibilities are avoided by buffering them between consecutive calibration stages.
• The workflow runs on, i.e., scales to, multiple subbands of the LOFAR test data set.
• The workflow is sufficiently documented, so that other teams and stakeholders can install the workflow and experiment with it using different configurations / algorithm settings.

We started with a detailed analysis of the Rapthor (LOFAR) workflow as it encodes the knowledge on how to tune the various steps in consecutive self-calibration cycles. This analysis is reported on Confluence.

Based on this, we developed a Python script performing the first three major self-calibration cycles. This does not only satisfy both key results, but also comprises a sufficiently substantial amount of computing that this workflow can be used for benchmarking in search of bottlenecks. Execution of more cycles will make run times longer thereby slowing down the cycle of implementation, testing, benchmarking and identification of bottlenecks / improvements.

These first three major cycles are described in detail in a progress diagram.

The code can be found in the git repo.

This workflow was demoed during System Demo 18.5. This ticks the first, third and fourth acceptance criteria.

To meet the second acceptance criterion, we extended the DPBuffer object to contain model visibilities as well. This enables passing along previously predicted model visibilities in-memory. As the model visibilities can now be stored in the DPBuffer object, no separate data structures were needed in DDECal, DP3's direction-dependent calibration function. This did not only simplify DDECal, but also resulted in fewer data copies, which lowered memory usage. As DDECal can also be called with a single direction, both KR1 and KR2 were achieved.

During System Demo 18.6, we also presented timing results. These showed that time is saved by avoiding multiple predicts of the same model visibilities without introducing (significant) overhead, implying that the benefit of avoiding unneeded predicts is almost 100%.