Integrate and demonstrate multi-node imaging distribution in Low and Mid AA2 self-calibration pipelines

Whilst the distribution of calibration stage has been demonstrated within both the Mid and Low pipelines this has so far exercised single node processing of the imaging and deconvolution stages. This work aims to demonstrate the ability to scale these latter stages by exercising multi-node distribution over ~10 nodes. This should be run as part of the self-calibration pipeline to understand overall performance and ability to scale.

There is related work to clarify the appropriate parameters for testing this capability (SP-4089), but such tests should include representative precursor data and simulated data that can be realistically extrapolated to AA2 scale in order to gauge expected performance.

We are now aiming to make the pipelines available to appropriate stakeholders which will require that they are versioned, documented and released with appropriate information for setup and usage (see the developer docs https://developer.skao.int/en/latest/tools/documentation.html#documenting-the-public-api and release management pages https://confluence.skatelescope.org/display/SE/Guidelines%3A+Release+Management+Process for project-wide guidance as a starting point).

See more details here: https://miro.com/app/board/uXjVN2k7zp0=/?moveToWidget=3458764579524561331&cot=14