Data processing pipeline development, enabling capabilities for AA2 onwards

Target is still primarily AA2 onwards - updated title to be more inclusive of bringing some development forward if needed.

For SDP

1. By Q1 2024 we need to know the performance of running a representative MID and LOW continuum imaging pipeline with a representative AA2 scale data set. This Epic is focussed on providing a direction-dependent calibration and imaging pipeline for Low and Mid which is distributed across at least 6-10 nodes aiming to enable processing of an AA2-scale dataset in an observation-equivalent time (e.g. in 4 hours for a 4-hr observation) with the purpose of understanding how we need to scale to achieve self-calibration at AA2 and beyond. See details of the scaling challenge here: https://confluence.skatelescope.org/pages/viewpage.action?pageId=200031968 

For PSS

1. By Q2 2024, we need to have demonstrated:
   • Real-time ingest of tied-array CBF voltages (Low) and total power (Mid) at AA2-scale parameters
   • Real-time processing of 3 beams/node with a single pulse search (SPS) pipeline implemented on a selected architecture, using AA2-scale Low and Mid data input and search parameters
   • Implementation of Fourier Domain Acceleration Search (FDAS) code on FPGA that is able to search through 500 DMs on 3 AA2 beams in real-time
   • Processing of 3 beams/node with a Fourier Domain Acceleration Search (FDAS) pipeline implemented on a selected architecture, using AA2-scale Low and Mid data input and search parameters
   • Implementations of SPS and FDAS on selected architectures that can be iterated beyond AA2 on next-generation hardware (of the same architecture) to meet AA4 performance and power consumption requirements
   • Efficacy of single pulse detection and frequency domain acceleration search with SPS and FDAS that meets L3 requirements

For PST

1. By Q1 2024, we need to have demonstrated:
   • Polyphase filterbank inversion that meets the signal fidelity requirements
   • Basic 'flow-through' mode capability of real-time recording of reduced tied-array beam data to disk
   • Real-time generation of QA metrics for the tied-array beam data that supports the development of Low and Mid Operator interfaces
   • Rudimentary functionality for real-time GPU processing of tied-array beam data at AA2-scale data rates

See also:

• Replace with info from Epic Refinement: https://miro.com/app/board/uXjVOpqHkm4=/?moveToWidget=3458764529351650563&cot=14
• Review and pull in some of the descriptions from SP-1320