Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1857

Address technical debt and failure modes for SDP AA0.5 visibility receive workflow (and assocated interface emulators)

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Must have
    • PI12
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      Data capture is one if not arguably the core function SDP must provide for AA0.5. As such making sure this workflow is robust, resilient, usable, well tested, and extensible by other developers is vital.

      Show
      Data capture is one if not arguably the core function SDP must provide for AA0.5. As such making sure this workflow is robust, resilient, usable, well tested, and extensible by other developers is vital.
    • Hide

      As discussed at PI planning:

      • Review and address key areas of technical debt in the visibility receive workflow.
      • Add automated integration testing for existing manually tested MeasurementSets and using python and kubernetes with services:
        • plasma_store
        • plasma-mswriter
        • emu-receiver
        • emu-sender
        • plasma-receive-integration-test
      • Complete tests to analyse failure modes such as
        • Resilience to dropped packets streamed from a CBF emulator.
        • Resilience to repeated long-running operations including chunking for writing to MS
      • Add automated test for auto-correlation only mode for MID PSI (Use MID PSI MeasurementSet?)
      • Add an easy-to-use runner script for manual testing
      • Add plasma receive workflow documentation (in CBF-Emulator repo)
      Show
      As discussed at PI planning: Review and address key areas of technical debt in the visibility receive workflow. Add automated integration testing for existing manually tested MeasurementSets and using python and kubernetes with services: plasma_store plasma-mswriter emu-receiver emu-sender plasma-receive-integration-test Complete tests to analyse failure modes such as Resilience to dropped packets streamed from a CBF emulator. Resilience to repeated long-running operations including chunking for writing to MS Add automated test for auto-correlation only mode for MID PSI (Use MID PSI MeasurementSet?) Add an easy-to-use runner script for manual testing Add plasma receive workflow documentation (in CBF-Emulator repo)
    • 2
    • 2
    • 20
    • 10
    • Team_YANDA
    • Sprint 5
    • Hide

      jupyter notebook exposes test interface
      https://gitlab.com/ska-telescope/sdp/ska-sdp-integration/-/blob/0d879dad9af57e82016eb6fc8dcc20c7bbf338af/tests/notebooks/sdp-receive-workflow-testing.ipynb
      sysdemo slides
      https://docs.google.com/presentation/d/113Uui2IwSvGKJBb6LiBp20e67GWiv-g4duq0MgfM4Qs/edit#slide=id.p

      RTD pages

      https://developer.skatelescope.org/projects/cbf-sdp-emulator/en/latest/example-plasma-client.html
      https://developer.skao.int/projects/sdp-dal-prototype/en/latest/package/usage.html#a-simple-processor
      https://developer.skao.int/projects/ska-sdp-science-pipelines/en/latest/workflows/vis_receive.html

      Show
      jupyter notebook exposes test interface https://gitlab.com/ska-telescope/sdp/ska-sdp-integration/-/blob/0d879dad9af57e82016eb6fc8dcc20c7bbf338af/tests/notebooks/sdp-receive-workflow-testing.ipynb sysdemo slides https://docs.google.com/presentation/d/113Uui2IwSvGKJBb6LiBp20e67GWiv-g4duq0MgfM4Qs/edit#slide=id.p RTD pages https://developer.skatelescope.org/projects/cbf-sdp-emulator/en/latest/example-plasma-client.html https://developer.skao.int/projects/sdp-dal-prototype/en/latest/package/usage.html#a-simple-processor https://developer.skao.int/projects/ska-sdp-science-pipelines/en/latest/workflows/vis_receive.html
    • 14.2
    • Stories Completed, Integrated, Solution Intent Updated, BDD Testing Passes (no errors), Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI24 - UNCOVERED

    Description

      For assessment of technical debt from PI11 see https://miro.com/app/board/o9J_lWqPSwQ=/?moveToWidget=3074457361392491628&cot=14

      This ticket picks up on what was done in PI11 in SP-1654.

      Intended to provide a test to work with SDP standalone in PI12, with an emulator for the CBF (send) functionality without emulating the CSP LMC (unless that makes things easier) - Probably best to void updating to new tango base classes in CBF emulator unless absolutely required.

      Other notes:

      • Need to test auto-correlation only mode for MID PSI testing: https://confluence.skatelescope.org/display/SE/PSI+MID+System+Integration+Test+1
      • At some point (possibly PI13) will likely need to revisit the concurrency of RT processing associated with visibility receive (eg using Plasma MS writer, flagging + RCAL pipeline, QA metric and displays, possibly fast imaging?)
      • Need to document solution architecture or make it easier to find how this workflow works with Plasma & sdp-dal
      • Need to better understand performance and failure modes for use of Apache Plasma for visibility receive

      Attachments

        Issue Links

          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1845 Further elaboration of acceptance tests for standalone SDP

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1546 Design and implement telescope model for visibility receive

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1654 Testing of the visibility receive workflow for AA0.5 use

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1845 Further elaboration of acceptance tests for standalone SDP

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1897 Demonstrate integration and configuration of initial QA metrics and display for visibility receive workflow

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-1839 Create releases for SDP components deployed with SKAMPI

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          mentioned on

          GitLab Commit - Merge branch 'sp-1857' into 'master'

          GitLab Merge request - SP-1857 Emu-Recv Testing Fixes

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                f.graser Graser, Ferdl
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete816.0
                  Total816.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel