Distributed Dask Performance

Follows work from SP-2086. Aims to address questions posed in goal SPO-1593

Who?

SDP pipeline developers, SDP architects

What?

• Identify and investigate the performance of Dask through focused testing using an i/o insensitive distributed processing pattern that targets one of the most challenging SDP imaging use cases.
  • This specifically means addressing the central scaling challenge: Keeping both image and grid data (and therefore visibilities once we get to gridding) distributed independently.
  • To prove this, we should show that we can effectively distribute both the image and subgrid load (i.e. demonstrate that memory requirements per node decrease as we add more nodes) - while still being able to correctly transform one into the other.
  • Note that checking correctness would generally require solving a full-size FFT here - it is therefore advisable to test with simple patterns where only a few (known) image or uv grid points are set in the input, as that means we can relatively cheaply predict any output uv grid or image point by direct evaluation of the Fourier transformation.

Why?

In the not too distant future (ideally by the end of PI14) we need to make a pivot or persevere decision on whether Dask will be used as an EF for SDP development or will we need to focus on other options. Dask is an attractive option since it has great community support and may improve interoperability with other astronomy software outside of SKA. We do believe there may be a number of problems with Dask scaling, and investigating is clearly very important if we are to adopt Dask for most of the SKA pipeline development for the foreseeable future.