Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3896

Design & prototype an automated solution for AA0.5 station beam calibration

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
    • 2
    • 2
    • 0
    • Team_YANDA
    • Sprint 5
    • Hide

      MCCS sky modelling and calibration tools have been incorporated into the new ska-low-mccs-calibration python module, which is available on gitlab and documented in the SKA developer portal. The main branch of the package contains all of the functionality developed for this feature. This includes:

      • Conversion between MCCS data products and SDP data models.
      • Automatic generation of sky models and model visibilities.
      • Automatic calculation and application of calibration solutions.
      • Access to new SDP solvers with improved polarisation calibration capabilities.
      • Generation of QA metrics.
      • Unit testing and documentation.

      Many of the functions in the module are based on existing station calibration scripts and have been updated to meet SKA software standards. They have also been adapted for more efficient processing in both runtime and memory usage. The interfaces to the main autonomous functions use SDP data models to minimise the coupling with intricacies of MCCS data products and the DAQ.

      The calibrate_mccs_visibility function can automatically form a sky model based on the contents of a Visibility Xarray Dataset. It returns calibration solutions, calibrated visibilities and QA metrics, and it is intended that this function will be called in an autonomous fashion from a MCCS script or pipeline.

      In the course of the PI, processing and design discussions were held with R.Subrahmanyan, Drew.Devereux, S.Asayama and M.Serylak, and a DP system demo was given. Plots and QA metrics presented in the demo were for the most part taken from Ravi's original notebooks, and some of these are returned by the calibrate_mccs_visibility function after calibration. Other metrics that indicate the state of the data and the sky model are also returned, however use of the package with more data of varying quality is needed to fine tune what is returned.

      Further details on the architecture of MCCS station calibration can be found here.

      Show
      MCCS sky modelling and calibration tools have been incorporated into the new ska-low-mccs-calibration python module, which is available on gitlab and documented in the SKA developer portal . The main branch of the package contains all of the functionality developed for this feature. This includes: Conversion between MCCS data products and SDP data models. Automatic generation of sky models and model visibilities. Automatic calculation and application of calibration solutions. Access to new SDP solvers with improved polarisation calibration capabilities. Generation of QA metrics. Unit testing and documentation. Many of the functions in the module are based on existing station calibration scripts and have been updated to meet SKA software standards. They have also been adapted for more efficient processing in both runtime and memory usage. The interfaces to the main autonomous functions use SDP data models to minimise the coupling with intricacies of MCCS data products and the DAQ. The calibrate_mccs_visibility function can automatically form a sky model based on the contents of a Visibility Xarray Dataset. It returns calibration solutions, calibrated visibilities and QA metrics, and it is intended that this function will be called in an autonomous fashion from a MCCS script or pipeline. In the course of the PI, processing and design discussions were held with R.Subrahmanyan , Drew.Devereux , S.Asayama and M.Serylak , and a DP system demo was given. Plots and QA metrics presented in the demo were for the most part taken from Ravi's original notebooks, and some of these are returned by the calibrate_mccs_visibility function after calibration. Other metrics that indicate the state of the data and the sky model are also returned, however use of the package with more data of varying quality is needed to fine tune what is returned. Further details on the architecture of MCCS station calibration can be found here .
    • 22.4
    • Stories Completed, Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI24 - UNCOVERED

    • Team_YANDA
    • Low G3

    Description

      See frame in PI21 Backlog board


      Beneficiaries (Who?)
      Science Commissioning and Science Operations

      Benefit Hypothesis (Why?)
      Once we get to 2+ stations, and eventually 6 for AA0.5, we need a way to automate station beam calibration solutions being generated for use by MCCS for providing the beamforming coefficients to the TPMs in SPS for forming station beams.

      Description (What?)
      Review and consolidate calibration scripts, creating a solution for automating the generation of calibration solutions and how they are captured/fed into the database of calibration solutions used by MCCS to provide coefficients for the TPMs (via the SPS HW interface).

      In PI21 this is expected to be a time-boxed activity, starting with reviewing and bringing existing calibration solutions in line with SKA standards and then, working towards a proof-of-concept/prototype for automated station beam calibration for AA0.5 (by or before PI23). This is limited to the generation of calibration solutions and does not include the direction-dependent beamforming coefficients.

      Acceptance Criteria (Success Criteria):

      • Review and agree on requirements for automated/online evaluation of station beam calibration solutions, including consideration for manual operator control of solution validity.
      • Take existing station calibration scripts and update to meet SKA software standards.
      • Work with MCCS to prototype a solution taking station visibilities produced by the DAQ and written to local storage and writing calibration solutions into the calibration database as an automated process. From initial discussions, MCCS will deal with the DAQ, local storage and the calibration database, while Yanda will take the in-memory visibilities and return in-memory gain solutions.
      • Working with Science operations and/or system science, begin to develop a set of heuristics / quality assessment metrics to display and assess calibration solutions.

      Attachments

        Issue Links

          Structure

            Activity

              People

                m.ashdown Ashdown, Mark
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete513.0
                  Total513.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel