AA0.5 real-time calibration pipeline for Mid

To satisfy a number of the AA0.5 commissioning scenarios , SDP will need to calculate complex gain corrections (Jones Matrices) for a single tied-array beam on boresight (in the direction of the centre of the primary beam) and provide these to the CBF beamformer at regular intervals (i.e. in near-real-time) during an observation. The scenarios show that cross-polarisation terms need to be solved for A working prototype of a real-time calibration pipeline for AA0.5 that can be tested on simulations will enable us gain insight into how to operate the pipeline and provide the gain solutions. This will inform the development of an integrated RCAL pipeline in future PIs. The role of RCal is to calculate the "G" and "T" terms only.

Working prototype of real-time calibration pipeline with documentation in an SKA GitLab repository. Simulations of AA0.5 data sets for testing the pipeline. A report on the results obtained with pipeline and any issues that need to be explored. Stretch: investigation of how the pipeline can be scaled up for later array assemblies.

For this task, we used the existing RCAL pipeline, which is implemented in RASCIL: https://gitlab.com/ska-telescope/external/rascil-main/-/blob/master/rascil/apps/rascil_rcal.py We have discussed the MID Commissioning Scenarios with the System Science Team and agreed to generate simulates which would be used for testing RCAL doing bandpass and gain calibration as well as RFI flagging. The discussion is documented here: https://confluence.skatelescope.org/display/SE/Commissioning+Scenarios+Discussion+for+MID+AA0.5 Simulated data: We simulated data sets for testing RCAL. These are documented and uploaded to Google Cloud Platform: Bandpass: https://confluence.skatelescope.org/display/SE/Simulations+for+testing+bandpass+calibration with data at https://console.cloud.google.com/storage/browser/ska1-simulation-data/simulations/rcal_mid_B2_SP-2899 (note that these data are not for AA0.5; all the other data sets are) https://confluence.skatelescope.org/display/SE/Simulation+of+data+at+AA0.5+scale+for+bandpass+calibration with data at https://console.cloud.google.com/storage/browser/ska1-simulation-data/simulations/rcal_mid_aa05_SP-2899/bandpass-cal-ORC-1450 Gain: https://confluence.skatelescope.org/display/SE/Simulation+of+data+at+AA0.5+scale+for+gain+calibration with data at https://console.cloud.google.com/storage/browser/ska1-simulation-data/simulations/rcal_mid_aa05_SP-2899/gain-cal-ORC-1424;tab=objects RFI: https://confluence.skatelescope.org/display/SE/SKA+Mid+RFI+Simulations See towards the bottom of the page section: "Results of RFI Simulations For agreed commissioning scenarios (AA0.5)". With data at  https://console.cloud.google.com/storage/browser/ska1-simulation-data/simulations/rcal_mid_aa05_SP-2899/RFI-sim-ORC-1425;tab=objects RCAL pipeline investigations: Data Requirements: We used the above data sets to test RCAL and note its usage, behaviour and additional improvement items. We documented results of both bandpass, and gain calibration runs, as well as RFI flagging: Bandpass: https://confluence.skatelescope.org/display/SE/Data+requirements+testing+for+bandpass+calibration+with+RCAL Gain and RFI flagging: https://confluence.skatelescope.org/display/SE/Data+requirements+testing+for+gain+calibration+with+RCAL The above pages explain the approach we took in testing the kind of data RCAL needs to provide acceptable calibration solutions and show the results. Computational Requirements: We also tested the pipeline focusing on the computational resources it needs to run. The tests and results are summarized here: https://confluence.skatelescope.org/display/SE/RCAL+Performance+Assessment Improvement suggestions: Finally, we summarized a set of improvement items which would allow RCAL to perform calibration in a manner that it is expected from it. We took into account the results of the discussion documented on SPO-2008 in a comment. The results are here: https://confluence.skatelescope.org/display/SE/Proposed+improvements+to+RCAL+at+AA0.5+scale