Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-97

Scalability of SDP architecture

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
      • Set baseline approach for measuring and comparing EFs in both a strong and weak scaling regime.
    • 40
    • 0.2
    • Team_ORCA
    • Sprint 5
    • 8.5
    • PI24 - UNCOVERED

    Description

      Might be more of an epic than a feature... Carry out further testing of execution frameworks and SDP architecture as discussed SDP CDR. The goal would be to demonstrate that the remedies for the scalability problem proposed by the architecture are sufficient for SKA1.

      General approach is to split the problem:

      1. Processing blocks / data islands: Demonstrate creating data islands and matching them up with execution engines. Show that we can reach necessary storage throughput and scale it to SKA scale.
      2. Execution engines: Show that within a data island / execution engine we can solve data-intensive SDP tasks. Goal would be to prove that we can reach necessary storage, network and compute throughput on a data island scale. We should especially try to show that we have multiple options on the execution framework choice, as this is still a risky area due to the tension between modifiability and varied performance requirements.
      3. Stretch: Investigate scalability of the parts of SDP where we cannot perfectly split the problem in this fashion, and need to rely on Data Queues. Examples would be calibration and (possibly less problematic) quality assessment. We have to show that we can implement such functions in a meaningful way (i.e. deal with limited bandwidth and loose synchronisation).

      Attachments

        Issue Links

          Structure

            Activity

              People

                m.deegan Deegan, Miles
                m.bartolini Bartolini, Marco
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 40.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel