SDP workflows demonstrating performance at ~AA2 scale

Follows from SP-1308, where:

Who?

• SDP developers
• System scientists 
• Observatory Scientists 
• (Wider astronomy radio community)

What? (outcomes)

To be updated after the Processing CoP meeting on the 11/04/22: https://confluence.skatelescope.org/display/SE/2022-04-11+%28Sprint+14.2%29+Processing+CoP+Meeting+Notes 

• A curated set of simulated and precursor data sets, along with associated scripts and tools for testing pipelines at scales up to AA2, including control of relevant corruptions (eg. DDE effects)
• Set of science pipeline workflows capable of processing AA2
  • Parallelism of up to ~20-30 nodes, waking use of GPU accelerators where appropriate
  • Incorporating state-of-the-art algorithms for imaging and calibration
  • Consideration for data structures and formats, with I/O performance suitable for full-array scale processing
• Established route to integration into SDP system
• Work with the services ART to establish test infrastructure, platform, and tools (enabler work for AA2+)
• Work with SRCs to provide simulated SDP data products
• Introductory material which is accessible and publically available, to ensure discoverability and to engage with a wider user community

Why?

• The ramp-up in the scaling demands of later Array Assemblies (from AA2 onwards) means we need to develop workflows ahead of the roll-out plan, delivering a set of workflows capable of processing AA2 scale data in advance of AA2 (~2025)
• Delivering SKA software that matches or exceeds the scientific and compute performance of current solutions within the next ~1-2 years is important because it will:

• • Greatly derisk the delivery of SDP data processing software for AA2 onwards, building solutions we can iterate on as we scale solutions for full-array operation and deal with the challenges of processing SKA telescope data in an integrated telescope system.
  • Build engagement and feedback from a wider user community, allowing us to work with SRCs prototyping efforts to deliver SKA software that will meet the needs of SKA users

Old Notes:

Development .. should demonstrate and provide a test of the core qualities of the SDP architecture; Pipelines and workflows .. structured with well-defined levels of abstraction ... This should result in being able to successfully demonstrate reuse of lower-level processing components with the development of these shared between more than one agile team.

What?

Suggest "thermally limited continuum images" (cf AA0.5? Same? Remove thermal limited?)

Why?

Defines most of the requirements (cal accuracy for example), whist not going full bore (ie spectral line mode)

(also add epic template fields)

How?

Review listed issues to make a natural list of sub pipelines, which should have been explored before.

Issues listed by R.Dodson and all possibly fitting the What-section:

(Also needed for AA0.5?) 

• Telescope model
• Sky model
• Ungridding
• Gridding
• Deconvolution

(New stuff for AA2)

• IDG
• RFI (flagging) / RFI library
• post-correlator transient detection
• simulation inputs (atmospheres)
• Compression: Half precision - BDA - UVGridding
• MSv3 related issues
• ICAL and then RCAL
• Regularisation in calibration
• Performance/Benchmarking
• Beam library

Old comment was: Develop calibration and imaging pipelines targeting 'small' scale, ~AA2 data size, 5-30 nodes (accuracy requirements from themal limits)

 This matches the figure in the presentation (attached in comments below)

See Miro for details: https://miro.com/app/board/o9J_klpPAzA=/?moveToWidget=3074457350419707408&cot=12