Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3446

Provide direction-dependent calibration in the Mid imaging pipeline

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      Direction-dependent calibration is necessary to ensure the science capability of SKA imaging for a number of the Mid bands (potentially all). As a potentially computationally expensive and time-consuming step, ensuring that the Mid self-calibration pipelines provide this capability is crucial in order to ensure we can test and scale the pipeline appropriately for AA2+.

      Show
      Direction-dependent calibration is necessary to ensure the science capability of SKA imaging for a number of the Mid bands (potentially all). As a potentially computationally expensive and time-consuming step, ensuring that the Mid self-calibration pipelines provide this capability is crucial in order to ensure we can test and scale the pipeline appropriately for AA2+.
    • Hide
      • Implement direction-dependent calibration, i.e. predict and calibrate different parts of the sky separately
      • Provide results of test runs (with profiling information), test data and parameter settings used to demonstrate the direction-dependent pipeline in SI (https://confluence.skatelescope.org/display/SWSI/Scaling+progress+report)
      • Apply direction-dependent calibration to imaging, i.e. split image into facets and demonstrate that different calibration solutions can be applied to facets
      Show
      Implement direction-dependent calibration, i.e. predict and calibrate different parts of the sky separately Provide results of test runs (with profiling information), test data and parameter settings used to demonstrate the direction-dependent pipeline in SI ( https://confluence.skatelescope.org/display/SWSI/Scaling+progress+report ) Apply direction-dependent calibration to imaging, i.e. split image into facets and demonstrate that different calibration solutions can be applied to facets
    • Intra Program
    • 5
    • 5
    • 0
    • Team_HIPPO
    • Sprint 5
    • Hide

      The MID self-calibration pipeline was extended to use DDECal from DP3 and facets that are being processed by WSClean. Relevant data structures and code for handling facets were written to enable this functionality. At the moment, the DD MID pipeline creates square facets, assigns sources from the initial sky model and creates files required by the WSClean. The directional independent MID pipeline was also modified to extend the number of parameters the user can adjust for feature SP-3477 and for benchmarking relevant to KR4. In parallel, the development on a code that creates facets based on Voronoi tessellation was written. Details are on the confluence page https://confluence.skatelescope.org/display/SE/HIP-614%3A+Plotting+Voronoi+tessellation+diagrams+using+Scipy+data.
      The DD MID self-calibration pipeline was tested on a simulated AA2 dataset generated in PI18. At the moment, the DD MID pipeline creates a regular grid of square facets, matches sources from the initial sky model to their respective facets and from there provides the information required by DP3 and WSClean to perform direction-dependent calibration and imaging. The results are described on the confluence page https://confluence.skatelescope.org/display/SE/HIP-638%3A+Attempts+to+obtain+a+reasonable+image+of+the+AA2+Mid+field+with+DDECal. After some initial problems given by unfamiliarity with the software these problems were fixed and the pipeline is now producing reasonable results. 
      Documentation of the MID self-calibration was updated accordingly (https://developer.skao.int/projects/ska-telescope-ska-sdp-wflow-mid-selfcal/en/latest/pipeline_dd.html).

      Show
      The MID self-calibration pipeline was extended to use DDECal from DP3 and facets that are being processed by WSClean. Relevant data structures and code for handling facets were written to enable this functionality. At the moment, the DD MID pipeline creates square facets, assigns sources from the initial sky model and creates files required by the WSClean. The directional independent MID pipeline was also modified to extend the number of parameters the user can adjust for feature SP-3477 and for benchmarking relevant to KR4. In parallel, the development on a code that creates facets based on Voronoi tessellation was written. Details are on the confluence page https://confluence.skatelescope.org/display/SE/HIP-614%3A+Plotting+Voronoi+tessellation+diagrams+using+Scipy+data . The DD MID self-calibration pipeline was tested on a simulated AA2 dataset generated in PI18. At the moment, the DD MID pipeline creates a regular grid of square facets, matches sources from the initial sky model to their respective facets and from there provides the information required by DP3 and WSClean to perform direction-dependent calibration and imaging. The results are described on the confluence page https://confluence.skatelescope.org/display/SE/HIP-638%3A+Attempts+to+obtain+a+reasonable+image+of+the+AA2+Mid+field+with+DDECal . After some initial problems given by unfamiliarity with the software these problems were fixed and the pipeline is now producing reasonable results.  Documentation of the MID self-calibration was updated accordingly ( https://developer.skao.int/projects/ska-telescope-ska-sdp-wflow-mid-selfcal/en/latest/pipeline_dd.html ).
    • PI22 - UNCOVERED

    • SDP-G2

    Description

      Direction-dependent calibration is necessary to ensure functional completeness of the Mid imaging pipeline. The most challenging Mid observations to process will be at the lowest frequencies where diffraction is large and the ionospheric corrections will need to be calculated in a direction-dependent way. To understand the scaling challenge of the mid pipeline, a representative example of such processing must be developed. This will continue the work on the Mid imaging pipeline which currently provides direction-independent calibration.

      This likely requires moving to the use of DP3 direct prediction and ideally would incorporate and build on the ability to do in-memory data exchange between DP3 stages in order to link different processing functions and avoid I/O overhead whilst adding the direction dependent capability (as  developed in SP-3447).

       

      Who?

      • SDP development teams, commissioning scientists, Telescope operators.

      What?

      • Require a self-calibration workflow capable of direction-dependent calibration for SKA1 Mid

      Why?

      • While direction-dependent calibration is more critical for SKA Low, low bands of SKA Mid will also require it (at minimum).
      • This feature introduces significant pipeline complexity, and is therefore important to introduce early

      Attachments

        Issue Links

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                D.Fenech Fenech, Danielle
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 5.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1946.0
                  Total1946.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel