RCAL integrated as an SDP workflow

For the AA0.5 system, a real-time flagging and calibration workflow will either be a required SDP function or, at the very least be extremely valuable, to have for testing to reduce the system risk of operating this workflow which needs to exercise interfaces with multiple SKA software systems and provide a solution for a single bore-sight tied-array beam which may prove valuable for engineering and science commissioning tests.

This functionality is anticipated to take several PIs to complete and test to provide a robust and resilient workflow for use with AA0.5 commissioning tests, continuing this development is therefore crucial at this stage.

• Integrated SDP workflow which flags and calibrates simulated data.
• Investigation of how the workflow can be tested in SDP and System integration environments.
• Investigation of how calibration solutions can be fed back to CBF.
• Improved version(s) of the real time flagging algorithm resulting from the analysis and comparison.
• A report on the performance comparison of the different flagging algorithms.

HIPPO Team

The sum-threshold RFI flagger was merged into the main branch of the processing function library and integrated into the RCAL workflow (processing script). The real-time RFI flagger was improved and tested on real and simulated data. A two-state algorithm was implemented into C with the expectation that it will be merged into the processing function library in PI15. The confluence page for RFI was amended (https://confluence.skatelescope.org/display/SE/Exploring+statistical+approaches+to+measure+variations+of+RFI+contaminated+data).

Orca Team

We have implemented a prototype of an integrated receive and real-time calibration pipeline. The calibration has been implemented as a visibility "consumer" which can be driven by the receive process as part of the existing visibility receive workflow. The code is in the RASCIL repository in the yan-982-rcal-consumer branch. It has not been merged yet because there are outstanding questions to resolve, as described in the related merge request. Due to outside factors, our work on the integration was started until much later than planned. As a result, it is only capable of processing a single time sample at a time. It was not possible to integrate it with the flagging because that would require more than one time sample to be held in memory.

We have investigated the interface for publishing real-time calibration solutions to be consumed by CBF. The current versions of the ICDs are lacking in detail, so we have raised ADR-58 to clarify the issues, including the the dimensions, sizes and data rates.