Enable Mid global pointing calibration and holography

Enable commissioning scientists to carry out observations with SKA-Mid to calibrate the global pointing model of the dishes and map their beams, and to extract the data to analyse offline. In the process it will mature our understanding and the implementation of the control system interfaces to acquire pointing data from the dishes, and the interfaces and procedures for extracting data products following a commissioning observation.

1. The appropriate SDP data products from a series of reference pointing observations (pointing offsets with the the appropriate metadata, and possibly the raw visibilities too) can be extracted for offline analysis to construct a global pointing model. - ORCA
2. The SDP is capable of capturing visibility data with the appropriate metadata including pointing to be used for offline holography analysis with KatHolog, and the data sets can be extracted. (Note the offline analysis might require some post-processing to convert data formats.) - YANDA (maybe not KatHolog bits)
3. Convert the output MeasurementSet to KatHolog format - ORCA
4. Simulated data sets have been made available to test the SDP part of the global pointing calibration and holography observations (both TBC). - ORCA (reference pointing output HDF5 files for given sources)

Releases:

• Pointing offset calibration pipeline: 0.7.0
• pointing-offset script: REL-1681

AC1:

• We investigated how to use the Data Product Dashboard API for downloading data and documented it on Confluence: https://confluence.skatelescope.org/display/SE/Access+pointing+data+from+Dashboard+API 
• We have updated the pointing data access Jupyter Notebook to incorporate downloading the data: https://gitlab.com/ska-telescope/sdp/ska-sdp-notebooks/-/blob/main/src/ska-sdp-find-pointing-outputs.ipynb 

AC2: - 

AC3:

• Notebook provided for converting between SKA MS and KatHolog data formats. It is linked to the following page, which also describes the data KatHolog needs where we store that at the moment: https://confluence.skatelescope.org/display/SE/KatHolog+Data+Format This was shared with Mattieu De Villiers

AC4:

• We generated HDF5 pointing data products for 50 sources from input data provided by System Science in the form of a TPOINT file:
  • data products are available on GDrive: https://drive.google.com/drive/folders/1CW6o8lvwAmH-NLvyQvBmsrYhTq44fbi9?usp=drive_link
  • python script to generate the data has been added to ska-sim-mid: MR74, MR76
  • ORC-2012 describes the TPOINT file obtained from System Science and links the data.
  • Jira tickets describe feedback we got from Justin Jonas and the process of conversion from TPOINT to HDF5: ORC-2043, ORC-2070, ORC-2087 

Additional work

• Solution Intent pages about the pointing pipeline and its data product have been added: 
  • https://confluence.skatelescope.org/display/SWSI/The+Pointing+Offset+Calibration+Pipeline 
  • https://confluence.skatelescope.org/display/SWSI/Pointing+Offset+Data+Product
• SDP pointing-offset notebook updated: MR60 (to be merged once vis-receive is released)
• SDP pointing integration test updated: MR359. This uses new simulated data and provides proper asserts of the output data
• Work to resolve problem with processing MS written by mswriter of vis-receive has progressed:
  • Further exploration of what may cause the problem: https://confluence.skatelescope.org/display/SE/PI23.2+investigation (and sub-pages)
  • ska-sim-mid supports using TelModel for antenna layouts:
    • ska-sdp-datamodels MR64 
    • ska-sim-mid MR72
  • New simulations were generated to fix some of the problems: https://console.cloud.google.com/storage/browser/ska1-simulation-data/simulations/pointing-offset/orc-2052 (see also ORC-2052). Note that these still need reordering by Antenna for the processing to work
• We tested the interfaces between Dish LMC and SDP for obtaining achieved pointing data: https://confluence.skatelescope.org/display/SE/Testing+with+Dish+LMC?src=jira We didn't note any problems that would require immediate fix. We will need real data to test with in order to see if any updates are needed. As it stands, the interface works.
• Pointing pipeline was updated to give the option to fit the beams to parallel-hands gains: MR82