Demonstrate visibility streaming pipeline with distributed facets

• Scaling strategy implemented in AA2 pipelines might not be enough to cover AA* Mid ICAL use cases
• We also need to evaluate technology options
• on the storage side (the existing measurement set implementation is a known and recurring bottleneck, even though it likely won't matter until large scales)
• on execution frameworks and networking (we have demonstrated that Dask can orchestrate pipeline, but can it provide the throughput necessary?)
• portability to compute platforms (e.g. accelerators)

Who?

• Pipeline developers
• System scientists (Commissioning)
• Commissioning and Operations staff planning for science commissioning & verification ahead of and during AA2

• Implement performance prototype that implements an imaging major loop (from model images to dirty image) using visibility streaming while holding facet data in-memory in a distributed fashion. This will require the following components:
• A way to generate grid data in a distributed fashion. Obvious option would be to start with the "distributed Fourier Transformation / SwiFTly" implementation [PANDO]: will implement MVP that does gridding of visibility partitions]
• Match subgrid regions with visibility partitions streamed from storage [PANDO]
• Processing functions
  • Degridding corrections to facet
  • Degridding of visibility partition: Nearest-neighbour unless CPU gridder is provided early PI22 [PANDO]
  • Subtraction of visibility partition
  • Gridding of visibilities: Nearest-neighbour unless CPU gridder is provided early PI22 [{*}PANDO]{*}
  • Gridding corrections to facet