Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2484

Data Models (structures) for use with the Processing Function Library

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Must have
    • PI15
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      (see why in description)

      Show
      (see why in description)
    • Hide
      • Document all existing data models used in the processing function library (visibilities, uvws, images, source models...). Socialise as part of the processing CoP.
      • Internal refactoring to simplify using data structures and ensure consistency (try to eliminate magical constants)
      • Demonstrate value from refactored data models by adding utility processing functions (to be discussed - weighting, phase rotation...?)
      Show
      Document all existing data models used in the processing function library (visibilities, uvws, images, source models...). Socialise as part of the processing CoP. Internal refactoring to simplify using data structures and ensure consistency (try to eliminate magical constants) Demonstrate value from refactored data models by adding utility processing functions (to be discussed - weighting, phase rotation...?)
    • Intra Program
    • 3
    • 3
    • 0
    • Team_HIPPO
    • Sprint 3
    • Hide

      The data models used within the Processing function library (PFL) are accessible in the PFL documentation here (https://developer.skao.int/projects/ska-sdp-func/en/latest/data_models.html). The processing function library was refactored to use these models on interfaces, and the models used were presented and discussed on multiple instances of Processing CoP. Data models were presented on processing CoP 2022-06-27(Sprint 15.1) (https://confluence.skatelescope.org/display/SE/2022-06-27%28Sprint+15.1%29+Processing+CoP+Meeting+Notes). Discussion followed with particular focus on the question of Sky Coordinates and suitable representation of these within data models. Based on discussions (thanks to Andreas Wicenec and Tammo Dijkema) the SkyCoord object was implemented into PFL (https://developer.skao.int/projects/ska-sdp-func/en/latest/sky_coord.html). This is compatible with SkyCoord from astropy and astropy SkyCoord object can be used as an argument in the PFL interfaces (discussion on what type of coordinates will be used by PFL is required).
      A phase rotation processing function was added that demonstrates the usage of the data models (https://developer.skao.int/projects/ska-sdp-func/en/latest/proc_func_phase_rotation.html).

      Show
      The data models used within the Processing function library (PFL) are accessible in the PFL documentation here ( https://developer.skao.int/projects/ska-sdp-func/en/latest/data_models.html ). The processing function library was refactored to use these models on interfaces, and the models used were presented and discussed on multiple instances of Processing CoP. Data models were presented on processing CoP 2022-06-27(Sprint 15.1) ( https://confluence.skatelescope.org/display/SE/2022-06-27%28Sprint+15.1%29+Processing+CoP+Meeting+Notes ). Discussion followed with particular focus on the question of Sky Coordinates and suitable representation of these within data models. Based on discussions (thanks to Andreas Wicenec and Tammo Dijkema) the SkyCoord object was implemented into PFL ( https://developer.skao.int/projects/ska-sdp-func/en/latest/sky_coord.html ). This is compatible with SkyCoord from astropy and astropy SkyCoord object can be used as an argument in the PFL interfaces (discussion on what type of coordinates will be used by PFL is required). A phase rotation processing function was added that demonstrates the usage of the data models ( https://developer.skao.int/projects/ska-sdp-func/en/latest/proc_func_phase_rotation.html ).
    • 18.2
    • Stories Completed, Integrated, Solution Intent Updated, BDD Testing Passes (no errors), Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI24 - UNCOVERED

    • SPO-1784

    Description

      Who?

      • SDP workflow developers

      What? (outcomes)

      • Set of well defined and documented data models (structures) for use with SDP Processing Function Library (PFL) methods for Self-calibration and Fast Imaging workflows.

      Why?

      • Processing Function Library (PFL) methods need clean portable interfaces but in order to build workflows
      • This will also enable definition of higher-level data structures to manage data in workflows, as well as de/serialisation from/to storage.

      Other considerations:

      • Future proofing - can the chosen data models represent everything we will want to pass between processing functions?
      • Dependency on existing PFL and related developments and development of initial Self-cal and fast imaging workflows
      • Consideration for C, C++ and Python bindings. Try to keep within general meta-"data model" of primitives, tensors (i.e. arrays) and possibly later tables.
      • Consideration for use with existing workflows and data reduction packages & applications
      • Consideration to load / save (deserialise/serialise) to storage/disk (possibly also important for testing)

      References

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete77.5
                  Total77.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel