Integrate, refine and test real-time pipelines

AC: New releases of real-time processing scripts which have been refactored to remove the need to specify internal configuration in parameters [ ORCA (pointing) ] New versions (0.4.0 both) of the pointing offset calibration pipeline and processing script have been released. A full list of updates can be found on the related ORCA objective: SPO-2918.  Configuration parameters of the pointing-offset script are in the documentation . The script itself configures the QueueConnector device, which the user does not need to worry about.  AC: Interfaces for publishing calibration solutions are agreed (ADRs decided).   [ ORCA ( ADR-76 ) ] We have attended and contributed to the discussions about ADR-76 on the Arch Sync meeting in January. We agreed on the interface going from SDP to TMC, and most of the issues around TMC to SDP were also discussed. We believe this ADR has been decided, however, during the last couple of days, we found out that on the OMC side this is not the case. We should aim to resolve any outstanding questions in the next week or so. AC: Mid pointing calibration pipeline has been demonstrated to work for AA0.5 use cases. [ ORCA ] We generated simulated data for Mid Band 2, AA0.5 layout for 16k channels. The data can be found on GCP . Then we ran the pointing pipeline, testing with different frequency averaging (in terms of number of chunks). The results are discussed on Confluence: https://confluence.skatelescope.org/display/SE/Performance+test+for+MID+Band+2+AA0.5+use+case   In addition, we developed a script to generate Band 5 simulations. This we aim to use next PI for further testing and to generate the AA0.5 dataset for Band 5: MR for ska-sim-mid , Confluence page discussing the results. Two-dish pointing: [ ORCA ] We investigated how we could generated simulated data for the future development of the two-dish pointing mode. This is discussed here: https://confluence.skatelescope.org/pages/viewpage.action?pageId=261849543     Yanda Progress update 11/01/24: The receive pipeline has been enriched so that the receive process can signal processors, via Plasma RPCs, when an SDP scan has started and finished (although the actual wiring of the Scan/EndScan commands triggering these RPCs hasn't been implemented yet). This new RPC in turn is used by the mswriter processor to close the Measurement Set. This is covered by YAN-1605, YAN-1607, YAN-1610, YAN-1613 and YAN-1614. Work has started to: a) emit an appropriate event so other parts of the system can react to MSs being written (YAN-1606), and b) make the Scan/EndScan commands actually trigger the new RPCs. To further support for the pointing offset pipeline, the mswriter processor can now receive the SOURCE_OFFSET coordinates instead of calculating them internally (YAN-1615), while the mock dish devices will provide such attribute (still in progress). Additioanlly, the mock dish devices can now read single pointing table HDF5 files (YAN-1620). NALEDI QA calibration plots: QA plots are available for calibration outputs. (NALEDI) plots for displaying the fitted params from the pointing offset data have been created. Antennas that return rejected pointing offsets are shaded in red to indicate their exclusion. These have not been fully integrated into the QA display as of yet. The current version of the pointing offset tests in the SDP integrations repo does not write any data to the pointing_offset kafka topic that the QA data api is listening to. https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-display/-/merge_requests/143 https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-display/-/merge_requests/146 https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-display/-/merge_requests/149 https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-data-api/-/merge_requests/107 plots for displaying the gain amplitude and phase from the bandpass gain calibration have been created. Currently, only data from the first channel is displayed, and further work will be carried out to make all the channel data available. These have not been fully integrated into the QA display as of yet. https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-display/-/merge_requests/145 https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-display/-/merge_requests/152 https://gitlab.com/ska-telescope/sdp/ska-sdp-qa-data-api/-/merge_requests/112 HIPPO PI21: The preprocessing pipeline was modified to use SKA Python visibility data models and integrated with the ingest processor. Modified version of the pipeline is in the repository https://gitlab.com/ska-telescope/sdp/science-pipeline-workflows/ska-sdp-wflow-mid-preprocess/-/tree/hip-431_preprocessing_pipeline/src?ref_type=heads . Possible ways how to integrate preprocessing pipeline into real-time calibration pipeline were also investigated. Details can be found on the confluence page https://confluence.skatelescope.org/display/SE/Integrating+RFI+flagging+into+real-time+calibration+pipeline .