Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2716

Python wrapper for DPStep

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Should have
    • PI16
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      DP3 offers advanced functionality that is field-tested in LOFAR production pipelines and, in some cases, not available otherwise. Making this functionality available to astronomers using SKA-LOW will be essential. Even commissioning scientists will benefit from a number of basic functions (flagging, DI calibration, averaging) provided by DP3.

      Show
      DP3 offers advanced functionality that is field-tested in LOFAR production pipelines and, in some cases, not available otherwise. Making this functionality available to astronomers using SKA-LOW will be essential. Even commissioning scientists will benefit from a number of basic functions (flagging, DI calibration, averaging) provided by DP3.
    • 4
    • 4
    • 0
    • Team_SCHAAP
    • Sprint 4
    • Hide
      • We have refactored all DPSteps such that they do no longer need an InputStep. This ensures that the DPSteps do not refer back to the original MeasurementSet to obtain the required meta-data. Instead, the DPSteps now fully rely on properly updated DPBuffer (data) and DPInfo (meta-data) objects, i.e., they retrieve data and meta-data from memory.
      • We have created Python wrappers for the StepFactory (factory function to create a DPStep), DPBuffer and DPInfo, so that DPSteps can be created and called from Python.
      • To illustrate the use of DPSteps from Python, we have developed a workflow script that performs direction-independent calibration from RASCIL.
      • We have created a binary wheel to make the Python wrappers pip installable.
      • DP3 provides a lot of functionality. Not all possible DPSteps and configurations of DPSteps are supported yet. However, as we have realised the underlying infrastructure, it should be straightforward to add additional functionality in the Python interface as we go along / as need arises.
      Show
      We have refactored all DPSteps such that they do no longer need an InputStep. This ensures that the DPSteps do not refer back to the original MeasurementSet to obtain the required meta-data. Instead, the DPSteps now fully rely on properly updated DPBuffer (data) and DPInfo (meta-data) objects, i.e., they retrieve data and meta-data from memory. We have created Python wrappers for the StepFactory (factory function to create a DPStep), DPBuffer and DPInfo, so that DPSteps can be created and called from Python. To illustrate the use of DPSteps from Python, we have developed a workflow script that performs direction-independent calibration from RASCIL. We have created a binary wheel to make the Python wrappers pip installable. DP3 provides a lot of functionality. Not all possible DPSteps and configurations of DPSteps are supported yet. However, as we have realised the underlying infrastructure, it should be straightforward to add additional functionality in the Python interface as we go along / as need arises.
    • 17.5
    • Demonstrated, Satisfies Acceptance Criteria
    • PI24 - UNCOVERED

    • DP-SDP: G3

    Description

      DP3 contains a number of highly useful processing steps that are needed to enable image reconstruction workflows to achieve high-quality imaging results. These processing steps are defined as DPSteps and include operations like

      • Direction-independent (DI) calibration, including frequency-based solution constraints.
      • Correction of visibilities for DI effects
      • Data averaging over time and frequency, including Baseline-Dependent Averaging (BDA)
      • Direction-dependent (DD) calibration with advanced features such as adjustable solution intervals over time, frequency and direction, application of (smoothness) constraints on the solution and ability to work on BDA-ed visibilities.
      • Combining antennas / stations into a 'superstation'.
      • Phase-shifting.

      A generic Python wrapper for the DPStep would facilitate integration of processing functions in DP3 in a RASCIL workflow.

      Who?

      • Commissioning scientist
      • Astronomer

      What?

      • Generic Python wrapper for a DPStep

      Why?

      • DP3 offers advanced functionality that is field-tested in LOFAR production pipelines and, in some cases, not available otherwise. Much of this functionality will be essential for astronomers using SKA-LOW. Even commissioning scientists will benefit from a number of basic functions (flagging, DI calibration, averaging) provided by DP3.

      Attachments

        Structure

          Activity

            People

              b.nikolic Nikolic, Bojan
              s.wijnholds Wijnholds, Stefan
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Feature Progress

                Story Point Burn-up: (100.00%)

                Feature Estimate: 4.0

                IssuesStory Points
                To Do00.0
                In Progress   00.0
                Complete1031.0
                Total1031.0

                Dates

                  Created:
                  Updated:
                  Resolved:

                  Structure Helper Panel