Distributed Low-pipeline

We need to be able to self-calibrate AA2-scale data (define what this means) efficiently and given the large datasets, this will require highly parallelised and distributed processing. In order to understand how we effectively scale (i.e. distribute the processing) we need to provide a pipeline that includes the minimal functionality required that reflects the computationally and I/O intensive parts of this processing. For the Mid calibration pipeline this includes DI calibration, DD calibration, sky model prediction, imaging (gridding/degridding) and deconvolution (optionally distributed imaging?).  

Current progress has provided this functionality for Low, with a recently established ability to use multiple nodes (SP-3446) by distributing over time (in calibration) and frequency (in imaging). Given the necessary lead-time to optimise and incorporate the full functionality eventually needed, it is important that we test this pipeline distribution as soon as possible to ensure this will scale as required and make improvements as needed.

In order to achieve this aim, in PI20 we need to test this pipeline over multiple (3-5) nodes with a the LOFAR test data as well as a Low representative (scaled-down for development) simulated dataset. It is also key that we can establish and understand the performance of this pipeline (both per node and over multiple nodes) so we are able to confirm that it scales as expected. As part of this work, we need to make  improvements/optimisations to the overall performance based on areas of the processing shown to be limiting performance. One primary area highlighted so far is the imaging stage for which intra- and inter-node improved distribution and performance should be investigated. Specifically for the current implementation with wsclean this could build on potential optimisations of the I/O usage within wsclean by memory-storage and re-use of information between sub-tasks.

The scope of this will need to be finalised fully during planning following discussions with the contributing teams, though should ensure that a specific parameter-set and dataset is agreed for any testing and performance comparisons and that contributions from different teams is captured in updated acceptance criteria.

As we are now at a stage of the development where we have a strong internal (multiple team contribution) and external (e.g. SRC) requests we need to provide pipeline releases, this work should also include minimal effort to establish a pipeline release artefact to be placed in the CAR.

Continuation of work in PI19 to demonstrate and assess the performance of the distributed Low self-calibration pipeline. Should consider:

• testing with representative simulated Low data as well as LOFAR test set currently being used
• Compared overall performance over single to several nodes to assess scaling effectiveness
• Assessment of distribution design