Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1331

Initial implementation of (continuum) imaging pipeline(s)

Details

    • Feature
    • Not Assigned
    • PI9
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      Creates a baseline for SDP continuum imaging workflow development that can be developed over time.

       

      Show
      Creates a baseline for SDP continuum imaging workflow development that can be developed over time.  
    • Hide
      • Workflow script(s) based on current solution/tools implementing a continuum imaging self-cal cycle (TBC following the outcome of SP-1326)
      • Documentation including:
        • A description of each of the pipeline steps/stages identifying data and metadata interfaces to each.
        • A description of how to obtain and run the pipeline with example (simulated data) which must be versioned, and easily available to all SKA SDP developers.
      Show
      Workflow script(s) based on current solution/tools implementing a continuum imaging self-cal cycle (TBC following the outcome of SP-1326 ) Documentation including: A description of each of the pipeline steps/stages identifying data and metadata interfaces to each. A description of how to obtain and run the pipeline with example (simulated data) which must be versioned, and easily available to all SKA SDP developers.
    • 6
    • 6
    • 20
    • Team_NZAPP, Team_SCHAAP, Team_SIM, Team_YANDA
    • Sprint 5
    • Show
      See Test section for the documentation of Low and Mid workflows: https://confluence.skatelescope.org/display/SE/Continuum+Imaging+Pipeline+Design+and+Implementation#ContinuumImagingPipelineDesignandImplementation-Tests
    • 10.4
    • Stories Completed, Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    Description

      <This ticket is expected to be a collaboration between 2 or more teams. While there is scope for decomposing this into multiple tickets during PI planning the intension is to keep the effort as aligned as possible.>

      Following the outcome of SP-1326, this is a timeboxed activity to develop one or more initial continuum imaging pipelines (workflow scripts) that should become the baseline for future SDP workflow development in the period of PI9-PI12 (Feature Set: SP-1308)

      In PI9 it is not anticipated that this workflow will integrate into the SDP operational system prototype. It is intended that this workflow is based on current state-of-the-art solutions/codes, following the approach that has already been started as part of the analysis of the SKA1 Instrumental Simulations initiative. When developing this workflow, it is important to consider the future development as an SDP architecturally compliant workflow; In particular, identifying well-defined levels of abstraction with strict interfaces between functional processing components and scalability and data distribution achieved through a two-tiered model where a high-level workflow tier orchestrates data movement and process execution, and a lower-tier supports potentially highly connected processing.

      While the exact scope of the pipeline should be agreed following the outcome of SP-1326, the default expectation is that this pipeline will consist of a self-cal loop similar to that described by Figure 8-19 SKA-TEL-SKO-0001075-01_DesignBaselineDocument. 

      It is important that this pipeline script is made available in SKA Gitlab. This should be linked to a description of the processing steps along with their interfaces which where possible should be mapped to SDP architecture modules and components (continuing work started in SP-1326). Documentation should be either on Confluence or, preferably in the form of readthedocs page(s) accessible via the developer portal (and included or referenced from the page structure produced for SP-1328). Instructions on how to obtain and run the pipeline(s) should also be provided, with sufficient detail to allow another SDP developer to download and run the pipeline (with consideration for availability and well-defined software versioning), including how to obtain input data and check that outputs match expected results (which might form the basis of testing in future PIs).

      It is also anticipated that in order to test the pipeline(s) generated by this ticket, additional simulations will need to be run with more complete coverage of the frequency band. This should be discussed as part of SP-1326 (if not before at PI planning) so that scheduling of dependencies can be properly managed and should follow previous SKA1 MID/LOW simulations (ie SP-1183, SP-1057, and SP-1056).

       

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                r.brederode Brederode, Ray
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 6.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1657.5
                  Total1657.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel