Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1601

Integrate LEAP with receive using processing function interface

    XporterXMLWordPrintable

Details

    • Enabler
    • Not Assigned
    • PI10
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      Apache Arrow/Plasma is a relatively new technology with the potential to signficantly increase data throughput in comparison to classic batch processing which is often limited by disk I/O. Trying an actual use case allows to verify the suitability of the technology and the visibility data model.

      Show
      Apache Arrow/Plasma is a relatively new technology with the potential to signficantly increase data throughput in comparison to classic batch processing which is often limited by disk I/O. Trying an actual use case allows to verify the suitability of the technology and the visibility data model.
    • Hide
      1. Integration of LEAP into Plasma receive workflow using processing function interface (i.e. ska-sdp-dal)
        1. Specifically visibilities received by the receive workflow can be passed to the LEAP calibration solver to produce solutions without having to be written to disk. Does not necessarily have to be streamed yet.
        2. It is suggested to use PlasmaStMan supporting the visibility model (ADR-21) and MS as needed
        3. test data include a MS from the Sim repo
        4. Stretch: Also put calibration solutions back into the Plasma store
      2. Deployment of workflow as per attached figure demonstrated, ideally again on PSI
      3. Document lessons learned with a focus on adequacy of technology and architecture; note any issues with input MS data
      4. Stretch: EAGLE workflow definition
      Show
      Integration of LEAP into Plasma receive workflow using processing function interface (i.e. ska-sdp-dal) Specifically visibilities received by the receive workflow can be passed to the LEAP calibration solver to produce solutions without having to be written to disk. Does not necessarily have to be streamed yet. It is suggested to use PlasmaStMan supporting the visibility model ( ADR-21 ) and MS as needed test data include a MS from the Sim repo Stretch: Also put calibration solutions back into the Plasma store Deployment of workflow as per attached figure demonstrated, ideally again on PSI Document lessons learned with a focus on adequacy of technology and architecture; note any issues with input MS data Stretch : EAGLE workflow definition
    • 2
    • 2
    • 5.5
    • Team_YANDA
    • Sprint 5
    • Hide

      sysdemo 10.6 slides:
      https://docs.google.com/presentation/d/1N81ZiBPeKFWxUsxk_mYH-KiCqtvjK_yXfRhZVCvsewU/edit?usp=sharing
      workflow:
      https://gitlab.com/calgray/plasma-leap/-/tree/master/plasma-leap-receiver
      code:
      https://gitlab.com/ska-telescope/icrar-leap-accelerate/
      https://gitlab.com/ska-telescope/plasma-storage-manager

      Show
      sysdemo 10.6 slides: https://docs.google.com/presentation/d/1N81ZiBPeKFWxUsxk_mYH-KiCqtvjK_yXfRhZVCvsewU/edit?usp=sharing workflow: https://gitlab.com/calgray/plasma-leap/-/tree/master/plasma-leap-receiver code: https://gitlab.com/ska-telescope/icrar-leap-accelerate/ https://gitlab.com/ska-telescope/plasma-storage-manager
    • 10.6
    • Stories Completed, Integrated, Solution Intent Updated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI22 - UNCOVERED

    Description

      Ingest Pipeline workflow demo using Plasma object store, ska-sdp-dal, cbf-sdp-emulator, casacore plasma-storage-manager and processing workload icrar-leap-accelerate.

      Attachments

        Issue Links

          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1551 First batch pipeline integration using processing function interface

          • Won't have (this time) - Items labelled as "Won't have", have been agreed by stakeholders as the least-critical, lowest-payback items, or not appropriate at that time. As a result, "Won't have" items are not planned into the schedule for the next delivery timebox.
          • Funnel
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                m.dolensky Dolensky, Markus [X] (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1126.0
                  Total1126.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel