Optimise DP3 and WSClean for frequency distribution

See frame in PI21 Backlog board

Who? (Beneficiaries)

• Pipeline developers.
• System scientists (Commissioning).
• Commissioning and Operations staff planning for science commissioning & verification ahead of and during AA2.

Why? (Benefit hypothesis)

• It is critical that we reach the end of PI21 with either a solid understanding of our scaling strategy for AA2, or the ability to make a reasoned pivot.
• Needs much more detail about drivers for reviewing at this stage.

What? (Acceptance criteria)

• By the end of PI21 we have a demonstrated performance and scalability of the Low self-calibration & imaging pipeline working on a small number of nodes (4-10 per self-calibration subband) using inter- and intra-node parallelism
  • Choice of algorithms, (multi-node parallelism using frequency chunks: SCHAAP; faceting with shared data reads: PANDO)
  • Bottlenecks & reason for bottlenecks - i.e. if we do not get roughly 10x speedup from running on 10 nodes, why not? (benchmarking: SCOOP, SCHAAP; reorder step in DP3: PANDO)
• We have documented for both DP3 and WSClean a "reasonable" extrapolation to what performance we would expect for Low/Mid AA2 ICAL (outcome of benchmarking: SCOOP, SCHAAP)
  • Using current state-of-the-art software,
  • Assuming future optimisations.