Show
Copied from Miro (see: https://miro.com/app/board/uXjVKZ900uo=/?moveToWidget=3458764589180836457&cot=14 ): We want our ICAL pipelines to be able to keep up with the Low and Mid telescopes at AA2 scale, i.e., these pipelines should take at most twice the duration of the processed observation. We are currently a significant factor away from that goal.
The most recent benchmarking results for the Low pipeline show that the workload for calibration is increasing each cycle due to the increasing complexity of calibration due to an improving sky model. This is confirmed by the timings from the Rapthor pipeline which also include a 4th and 5th cycle with the latter using 100% of the data instead of ~20% of the data. This means that calibration could well become the main bottleneck of the pipeline.
The most recent benchmarking results for the Mid pipeline show that the predict stage in WSClean does not scale to multiple node (it actually becomes slower). This needs to be investigated and addressed.