Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1859

Add a flexible gridder interface to Jacal

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Should have
    • PI12
    • COM SDP SW
    • Data Processing
    • Hide

      Development of existing state-of-the-art imaging pipelines will help inform SKA SDP Pipeline development. This enabler adds a flexible gridder interface to JACAL, allowing:

      • the various gridding and solving options of Yandasoft to be partitioned in novel ways.
      • analysis of bottlenecks and parallelism
      • future connection with alternative gridding components from other SDP workflows.
      Show
      Development of existing state-of-the-art imaging pipelines will help inform SKA SDP Pipeline development. This enabler adds a flexible gridder interface to JACAL, allowing: the various gridding and solving options of Yandasoft to be partitioned in novel ways. analysis of bottlenecks and parallelism future connection with alternative gridding components from other SDP workflows.
      • Demonstrated use of gridder (possibly using standard continuum imaging simulation data)
      • Relevant documentation including information on access and use
    • 3
    • 3
    • 3
    • 1
    • Team_YANDA
    • Sprint 5
    • Hide

      https://confluence.skatelescope.org/display/SE/YAN-753+Deployment+and+Development+of+DALIUGE+Graphs+with+JACAL+Components+-+A+refresher

      Pprocessing workflows CoP presentation:
      https://drive.google.com/file/d/17Ql9rBBEKP-FzsSpQvxqN3rLA3Rz4wNK/view?usp=sharing

      Use eagle.icrar.org to visualize graph ICRAR/EAGLE-graph-repo/examples/jacal_CalcNElight.graph
      The new gridders and accessors are used within component CalcNElight, available in this github palette; input parset for the demo graph; associated data; CalcNElight.cc. New parset parameter gridder.partitiontype can be set to "time" or "wsort", for time or w-term based data access respectively.

      Merge request for YAN-756: https://gitlab.com/ska-telescope/ska-sdp-jacal/-/merge_requests/21

      Merge request for YAN-757 and YAN-758: https://gitlab.com/ska-telescope/ska-sdp-jacal/-/merge_requests/22

      Show
      https://confluence.skatelescope.org/display/SE/YAN-753+Deployment+and+Development+of+DALIUGE+Graphs+with+JACAL+Components+-+A+refresher Pprocessing workflows CoP presentation: https://drive.google.com/file/d/17Ql9rBBEKP-FzsSpQvxqN3rLA3Rz4wNK/view?usp=sharing Use eagle.icrar.org to visualize graph ICRAR/EAGLE-graph-repo/examples/jacal_CalcNElight.graph The new gridders and accessors are used within component CalcNElight , available in this github palette; input parset for the demo graph; associated data ; CalcNElight.cc . New parset parameter  gridder.partitiontype can be set to "time" or "wsort", for time or w-term based data access respectively. Merge request for YAN-756: https://gitlab.com/ska-telescope/ska-sdp-jacal/-/merge_requests/21 Merge request for YAN-757 and YAN-758: https://gitlab.com/ska-telescope/ska-sdp-jacal/-/merge_requests/22
    • 12.6
    • Integrated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI22 - UNCOVERED

    • Jacal

    Description

      To meet the extreme data challenges of the SKA, SDP will need to partition data and processing in novel ways. The Yandasoft imager has been designed from the start for widely distributed processing, and as such is an important testbed for SDP. However, some of the early design decisions associated with data access limit the flexibility with which the data and processing can be partitioned.

      In this enabler we will extend the continuum imaging capabilities of JACAL – an integration of Yandasoft processing elements within DALiuGE – by adding a data accessor (and associated interfaces) that can partition a dataset between processing components in a more flexible way. The intention is that the new interface be compatible with other SDP gridding components in future PIs.

      Attachments

        Structure

          Activity

            People

              D.Fenech Fenech, Danielle
              d.mitchell Mitchell, Daniel
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Feature Progress

                Story Point Burn-up: (100.00%)

                Feature Estimate: 3.0

                IssuesStory Points
                To Do00.0
                In Progress   00.0
                Complete417.0
                Total417.0

                Dates

                  Created:
                  Updated:
                  Resolved:

                  Structure Helper Panel