Investigate functionality, performance and scaling of other existing pipelines

See frame in PI21 Backlog board

Who? (Beneficiaries)

• Pipeline developers.
• System scientists (Commissioning).
• Commissioning and Operations staff planning for science commissioning & verification ahead of and during AA2.

Why? (Benefit hypothesis)

• We need to ensure that we maintain a good idea of what the state-of-the art is in terms of radio astronomy pipelines.
• As the large data amounts will make it tricky to test externally, we will likely even need to make preparations for running external pipelines on the SDP system in order to do testing.

What? (Acceptance criteria)

• (Re-)establish "radar" of pipelines and technologies - idea is that we periodically check their relevance to the SKA. (DDFacet: HIPPO, ASKAPSoft: YANDA) Recommended engagement levels might be something along the lines of:
  • Hold (we don't currently think they have something to offer us, but track them in case something happens);
  • Assess (there are interesting aspects to it that we might want to adopt, so we'd look into the design / trial integration);
  • Trial (we actually do end-to-end runs with our datasets, and aim for loose integration into SDP as batch processing blocks);
  • Adopt (we fully integrate, and actively work on all qualities relevant to us).
• Do this for some reasonably broad categorisation:
  • Preprocessing/self-calibration/instrumental calibration/data product preparation pipelines, individual algorithms etc.?
  • Architectural qualities we are interested in would be - as usual - modifiability and scalability & performance,
  • Functional qualities as of SKA requirements
  • YANDA: Analysis of ASKAPSoft continuum imaging with distributed facets (scalability, performance and perhaps modifiability)
• Do deeper analysis / test runs of pipelines in "assess" or better.