Show
Captured from description on Miro (see: https://miro.com/app/board/uXjVKZ900uo=/?moveToWidget=3458764589411581527&cot=14 ):
We want our ICAL pipelines to be able to keep up with the LOW and MID telescopes at AA2 scale, i.e., these pipelines should take at most twice the duration of the processed observation.
Profiles of our runs show that we have largely successfully managed to distribute the expensive operations of ICAL. However, it looks like (now) a lot of time is spent in phases where not much parallel work is happening at all - in some cases it actually looks like we spend hours using just a few (or even just one) core on the master node. The net result is that we are likely only using <5% of the compute available to us.