Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1693

Onboard and test complete set continuum imaging pipelines using simulation data (without DD effects)

    XporterXMLWordPrintable

Details

    • Feature
    • Should have
    • PI11
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      If we develop a set of suitable initial continuum imaging pipelines to act as a baseline for future development, we will be able to test for performance and scalability in a consistent way which will ensure a solution that meets SKA needs.

      Show
      If we develop a set of suitable initial continuum imaging pipelines to act as a baseline for future development, we will be able to test for performance and scalability in a consistent way which will ensure a solution that meets SKA needs.
    • Hide
      Show
      Pipelines made available in a suitable repository ( https://gitlab.com/ska-telescope/sdp/ska-sdp-continuum-imaging-pipelines ) with appropriate instructions Documentation of tests performed in written in confluence, including datasets to be used. Documentation of test results and comparisons written in confluence.
    • 3
    • 3
    • 11
    • Team_NZAPP, Team_SCHAAP, Team_SIM, Team_YANDA
    • Sprint 5
    • Hide

      Mark Ashdown added a comment - 26/Aug/21 2:17 PM

      Outcomes from the Sim team:

      We led the discussions about the standard outputs from the imaging pipelines and how to handle the direction-dependent effects from beams. The outcomes are here:

      Continuum imaging standard outputs
      Use of time variable primary beams in the continuum imaging QA

      We have improved the performance of the the RASCIL continuum imaging pipeline by modifying it to output moment/Taylor term images. It can now produce larger images on the same computing resources.

      We have thoroughly investigated the scaling of the clean step in the RASCIL imager as a function of the number of facets and PSF support. The results from this investigation are in Investigate scaling of clean step in rascil imager (SIM-980/1040).

      RASCIL has been rerun to produce results from the continuum imaging data sets with 16k images for Mid and Low which are described in Second generation Continuum Imaging results.

      We have improved the performance of the spectral estimation in the continuum imaging QA pipeline.

      Show
      Mark Ashdown added a comment - 26/Aug/21 2:17 PM Outcomes from the Sim team: We led the discussions about the standard outputs from the imaging pipelines and how to handle the direction-dependent effects from beams. The outcomes are here: Continuum imaging standard outputs Use of time variable primary beams in the continuum imaging QA We have improved the performance of the the RASCIL continuum imaging pipeline by modifying it to output moment/Taylor term images. It can now produce larger images on the same computing resources. We have thoroughly investigated the scaling of the clean step in the RASCIL imager as a function of the number of facets and PSF support. The results from this investigation are in Investigate scaling of clean step in rascil imager (SIM-980/1040). RASCIL has been rerun to produce results from the continuum imaging data sets with 16k images for Mid and Low which are described in Second generation Continuum Imaging results. We have improved the performance of the spectral estimation in the continuum imaging QA pipeline.
    • 14.2
    • Stories Completed, Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI22 - UNCOVERED

    Description

      Following the development of several continuum imaging pipelines in PI10 (SP-1548, SP-1605) , this should establish the first performance comparison which includes processing and I/O performance as well as image quality metrics as available (initial discussion of these has been captured in Continuum sync meetings and solution intent) (TBD: with known DDEs). This includes the development of and comparison to community (software used in scientific publications) validated pipelines (e.g. CASA, wsclean, DP3 etc.). Instructions on how to perform these comparisons should be provided and ideally, teams involved should perform the running and analysis of a pipeline they have not written to 'test' the documentation. The results of this comparison should be documented.

      Attachments

        Issue Links

          preceded by

          Spike - A time limited investigation to get some knowledge SP-1326 Determine initial design for a continuum pipeline and datasets

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1548 Standardise testing of initial (PI9) SDP imaging pipelines

          • Not Assigned - Priority is not assigned yet.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1605 Continuum imaging QA

          • Not Assigned - Priority is not assigned yet.
          • Done
          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1778 Package and benchmark a representative sample pipeline

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Implementing

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1775 Development and comparison of imaging pipelines including primary beam correction

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                D.Fenech Fenech, Danielle
                f.graser Graser, Ferdl
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete3178.0
                  Total3178.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel