Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1551

First batch pipeline integration using processing function interface

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Won't have (this time)
    • None
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      To enable the intended flexibility in exchanging processing components between execution engines, we need to be able to interface with them using a common interface.

      Show
      To enable the intended flexibility in exchanging processing components between execution engines, we need to be able to interface with them using a common interface.
    • Hide
      • Existing pipeline changed so a processing function relevant to overall performance is called via ska-sdp-dal facilities (gridding, DFT?)
        • Note that some example bindings to nifty-gridder are already provided by ska-sdp-dal
      • Prototype how to maintain meta-data as part of the pipeline to both make development easy and minimise need for copies
      • Evaluate meta-data models proposed in ADR-21, add suitable call and data schemas to https://gitlab.com/ska-telescope/sdp/ska-sdp-dal-schemas
      • Stretch: Investigate overheads of going this route.
      Show
      Existing pipeline changed so a processing function relevant to overall performance is called via ska-sdp-dal facilities (gridding, DFT?) Note that some example bindings to nifty-gridder are already provided by ska-sdp-dal Prototype how to maintain meta-data as part of the pipeline to both make development easy and minimise need for copies Evaluate meta-data models proposed in ADR-21 , add suitable call and data schemas to https://gitlab.com/ska-telescope/sdp/ska-sdp-dal-schemas Stretch: Investigate overheads of going this route.
    • 4
    • 6.25

    • PI22 - UNCOVERED

    Description

      Sibling feature to SP-1551. By PI 12 we would like to be in a position where we can assess whether Apache Plasma using Arrow data models is a viable route for the interface between execution engines and processing components. Up to this point we have verified that it can be used for streaming visibility data, however we still need to look into whether we can really use it for calling processing functions.

      For this feature, we should develop a further proof-of-concept integration using Plasma/Arrow on top of the DAL prototype. The idea would be that we explore more deeply the issues with getting data out of execution engines / pipelines into external pipelines, especially:

      • How to manage memory and minimise copies
      • How to track metadata and keep it associated with bulk data (XArray)?
      • (possibly performance aspects, e.g. throughput and response times)

      Suggested processing function might be a direct fourier transform (depending on SP-1552) or gridding (more complex, but might lead to more insight).

      Attachments

        Issue Links

          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1601 Integrate LEAP with receive using processing function interface

          • Not Assigned - Priority is not assigned yet.
          • Done

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                r.brederode Brederode, Ray
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 4.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel