Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3686

Data moving challenge with Rucio

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Not Assigned
    • None
    • None
    • SRCnet
    • Hide

      Expanding the prototype DDM (distributed data management) network and constraining what's possible in the data size / frequency of transfer parameter space would be valuable information when it comes to software downselection; whereas missing functionality e.g. IVOA integration can be developed, the underlying DDM architecture and how fit for purpose it is is absolutely fundamental. We can mitigate this risk (which is arguably the biggest risk we have around DDM selection) by demonstrating that it "performs" well at the scales required.

      The operational aspect of conducting this data challenge will help inform the NFR evaluation of the respective DDM system.

      Conducting such a data challenge is one of the end goals of the corresponding epic. 

      Show
      Expanding the prototype DDM (distributed data management) network and constraining what's possible in the data size / frequency of transfer parameter space would be valuable information when it comes to software downselection; whereas missing functionality e.g. IVOA integration can be developed, the underlying DDM architecture and how fit for purpose it is is absolutely fundamental. We can mitigate this risk (which is arguably the biggest risk we have around DDM selection) by demonstrating that it "performs" well at the scales required. The operational aspect of conducting this data challenge will help inform the NFR evaluation of the respective DDM system. Conducting such a data challenge is one of the end goals of the corresponding epic. 
    • Hide

      AC: Confluence doc of working assumptions used and specifics of the testing to be conducted in a 24 hour period

      AC: 24 hour period of tests conducted, with outcomes documented on Confluence along with grafana dashboard snapshots

      Show
      AC: Confluence doc of working assumptions used and specifics of the testing to be conducted in a 24 hour period AC: 24 hour period of tests conducted, with outcomes documented on Confluence along with grafana dashboard snapshots
    • 22.5

    • PI23 - UNCOVERED

    • tests-compilation

    Description

      • Conduct something that might resemble a Day in the life of data moving in the SRCNet
      • Liaise with BO in Science Operations to inform the specifics of the tests conducted, they will also be informed by https://jira.skatelescope.org/browse/SP-3650.
        • This will include but isn't limited to number of files, frequency, file size, dataset size, number of sites
      • Write down list of working assumptions (that may have been acquired as described above), and move data around according to these assumptions 
      • There is also a preparatory/operational aspect to this - getting sites ready, notifying them of the challenge, maintaining services for the critical testing period. This will inform the NFR evaluation of the respective DDM system.

      Attachments

        Issue Links

          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3592 Rucio demo with scientific data products

          • Won't have (this time) - Items labelled as "Won't have", have been agreed by stakeholders as the least-critical, lowest-payback items, or not appropriate at that time. As a result, "Won't have" items are not planned into the schedule for the next delivery timebox.
          • Discarded
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel