Integrate Mid pointing calibration

Demo is planned for PI21. Partly demoed at PI20 Demo as part of Mid Goal2. Also demoed at DP System Demo 21.3 ( slides ) Related Releases (other than SDP 0.18.0): pointing-offset processing script 0.3.0 and 0.3.1 (difference is the use of the QueueConnector device) pointing offset calibration pipeline 0.3.0 AC1: Pointing offset calibration pipeline is fully integrated with the control system at the SDP sub-system level. SDP-based notebook with full integrated test: https://gitlab.com/ska-telescope/sdp/ska-sdp-notebooks/-/blob/main/src/ska-sdp-pointing-script.ipynb This includes downloading data, running the CBF emulator, and running code to simulate sending dish pointings. Updated documentation that explains how to run the same thing from ITango, and using the MockDish devices can be found here . Details of changes this PI can be found below. AC2: Integrated Mid software system (including TBC: OSO-scripting, TMC, Dish LMC and SDP) capable of executing pointing offset calibration observations. The integration has happened under the coordination of Adam Avison. It was demoed at PI20 Demo. We contributed to the notebook's SDP side of things. The notebook can be found here . The contribution is in the second half (the notebook itself is not finished yet).  AC3: A manual test of the integrated system working on a simulated five-point reference pointing observation, driven by a Jupiter notebook. Same points as AC1 Note: these tests are using Meerkat data and not simulated data. See explanation below at "Simulations". ------- Detailed updates from this PI Pointing offset script and integration: Script adds the receive_addresses attribute to its state which is used by the subarray device. It contains the pointing FQDN to inform TMC about where it can find the pointing results. This required updating multiple repositories:  Processing script: MR95 Telescope Model to include the `pointing_cal` key in the receive_addresses schema: MR132 SDP integration to provide system configuration information in the Config DB. This includes tango device names, which the script needs: MR246 Subarray device now correctly merged receive_addresses attributes from multiple processing blocks: MR75 Scripting library updated to not use ports and hosts when the pointing-offset script calls it: MR51 The script obtains the processing block ID of the vis-receive script (needed to find the directory where data are saved) via dependencies: MR94 , MR48 (ska-sdp-scripting) + updates made by Mark Ashdown on LMC and Processing Controller Script is updated to work with v0.3.0+ of the QueueConnector device, which allows configuring the device from multiple scripts. This way the pointing attribute is configured via the pointing-offset script directly: QueueConnector configuration added to the scripting library: MR52 ; also released ska-sdp-scripting 0.6.3 Processing scripts updated: MR106 Integration test in the SDP integration repository to run a full 5-point scan is being updated. Currently draft MR : MR264 Pointing offset calibration pipeline: A wrapper was added to the pipeline which keeps the pipeline running and ready for processing new sets of pointing scans: MR52 For this, we also updated the pointing-offset helm chart to deploy a Deployment instead of a Job: MR37 (ska-sdp-helmdeploy-charts). (Initial exploration can be found on Confluence .) Convert azimuth to cross-elevation before fitting to the primary beam. This allows an easy replace of what input data we use, when we will get values in xel-el in the SOURCE_OFFSET column of the Measurement Set. We also updated the pipeline to read the FIELD table instead of the SOURCE table (to comply with the mswriter): MR42 Option to read on-sky offsets in cross-elevation - elevation from SOURCE_OFFSET column: MR47 create_visibility_from_ms function is fixed and now we can process the data written by the MS writer in vis-receive: MR45 (ska-sdp-datamodels) Added functionality to convert PointingTable to JSON and set up the pipeline to optionally send the results to Kafka in JSON format: MR46 (ska-sdp-datamodels), MR55 Added various improvements, including how the fitting is done by using the weights from the gain calibration to calculate the standard deviation on the gain amplitudes: MR44 Simulations: Throughout the PI, we continued investigating what causes the discrepancies between pipeline outputs when simulated data are used and expected results from the simulations (pages are not linked in order, see titles): Confluence page 1 , Confluence page 2 , Confluence page 3 ,  We fixed multiple bugs: MR43 (ska-sdp-wflow-pointing-offset), MR44 (ska-sdp-func-python) We held a workshop with various stakeholders to discuss the status: Confluence page 1 ,  Confluence page 2   The final conclusion is that there are two main areas that cause the differences: The simulation script needs to use a Gaussian Voltage Patter so that we get a symmetrical beam: Confluence page and Voltage Pattern file . The pipeline and the simulation code use different tools for coordinate conversions. One uses katpoint, the other uses astropy. When katopint is replaced in the pipeline with astropy, we found that the results are as expected. However, this is only for testing purposes, until there is a decision on what tools to use in the final version:  At the moment we are not able to use the simulated data with the pipeline, because of the coordinate conversion problem, which we will need to address next PI (this requires discussions and decision on steps forward) - one option is that we only use simulated data when the pipeline reads on-sky offsets from the SOURCE_OFFSET column, in which case the pipeline itself doesn't need to do coordinate conversion; however, we would not be able to use simulations when either the DIRECTION or TARGET columns are used.