Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1023

Proof-of-concept of visibility streaming over apache arrow/plasma for real-time processing

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      Gaining some experience with memory data models and technologies can inform architectural choices in the memory management domain (presentation).

      Show
      Gaining some experience with memory data models and technologies can inform architectural choices in the memory management domain ( presentation ).
    • Hide
      • Visibilities streamed into plasma shared-memory database - ideally from an ingest workflow, but could be a simulator
      • Consumer/processor accepts visibilities from plasma store in real-time and does some processing - could be simple statistics/signal display
      • Consider logical data model
      • Stretch: Investigate providing meta-data alongside visibilities (e.g. uvw / antenna coordinates)
      • Stretch: Have a look at performance (latency, throughput)
      • Stretch: Support streaming to multiple consumers, possibly changing at runtime ("simulate" multicast)
      Show
      Visibilities streamed into plasma shared-memory database - ideally from an ingest workflow, but could be a simulator Consumer/processor accepts visibilities from plasma store in real-time and does some processing - could be simple statistics/signal display Consider logical data model Stretch: Investigate providing meta-data alongside visibilities (e.g. uvw / antenna coordinates) Stretch: Have a look at performance (latency, throughput) Stretch: Support streaming to multiple consumers, possibly changing at runtime ("simulate" multicast)
    • 3
    • 3
    • 2
    • Team_YANDA
    • Sprint 5
    • Hide
      Show
      Lessons learned: https://docs.google.com/document/d/1pscVbF5l-0TbLmWrP0wCGQDodDzfpM4ghoJzF8wBj_A/edit Review and Requirements: https://confluence.skatelescope.org/pages/viewpage.action?pageId=113803291 SDP-DAL usage description: https://confluence.skatelescope.org/pages/viewpage.action?pageId=113810294 BDD test https://jira.skatelescope.org/browse/XTP-958 Sprint review 7.3, demo recording: https://drive.google.com/file/d/1D-mLMyF-lKKLFMpN3Nax3QL_0qotlLQT/view?usp=sharing Sysdemo 7.5: https://confluence.skatelescope.org/display/SE/2020-08-12+DP+ART+System+Demo+7.5
    • 7.5

    • PI24 - UNCOVERED

    • Team_YANDA
    • SPO-575

    Description

      This is follow-up work to the DAL efforts started in -SP-665-, which resulted in an Apache Arrow/Plasma-based prototype in https://gitlab.com/ska-telescope/sdp-dal-prototype

      https://developer.skatelescope.org/projects/sdp-dal-prototype/en/latest/README.html

      While the technology choices are not final, it would be good to gain some experience with this type of approach. The aim for this feature is stream visibilities as it would happen in an SDP ingest pipeline. The reason for targeting this use case first is because we already have multiple teams providing receive and (to a lesser degree) real-time processing capabilities, and we need a way to integrate them.

      Presentation with background on the prototype:

      https://docs.google.com/presentation/d/1Sc2Z_7FZBuSAIjz7qapQ8S0KnApLo6_m2cHttNvqm4k

      Attachments

        Issue Links

          is required by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1155 Complete integration of visibility receive workflow

          • Not Assigned - Priority is not assigned yet.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1178 Test updated visibility receive code with preallocated nodes

          • Won't have (this time) - Items labelled as "Won't have", have been agreed by stakeholders as the least-critical, lowest-payback items, or not appropriate at that time. As a result, "Won't have" items are not planned into the schedule for the next delivery timebox.
          • Discarded
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-950 Updated visibility receive workflow integrated into the SDP prototype

          • Not Assigned - Priority is not assigned yet.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 3.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete824.0
                  Total824.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel