Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2446

Support replication of a prototype SDP->SRC data transfer with Rucio

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Should have
    • PI14
    • None
    • None
    • Data Processing
    • Hide

      See description

      Show
      See description
    • Hide

      KR1: NALEDI team are supported to deploy rucio-analysis tests tuned to show metadata-based subscription behaviours transferring data products from IDIA site to European sites.

      KR2: Onboarding guide developed to support NALEDI and future teams.

      Show
      KR1: NALEDI team are supported to deploy rucio-analysis tests tuned to show metadata-based subscription behaviours transferring data products from IDIA site to European sites. KR2: Onboarding guide developed to support NALEDI and future teams.
    • Intra Program
    • 0.5
    • 0.5
    • 5
    • 10
    • Team_ESCAPEES
    • Sprint 2
    • Hide

      Successful subscription based transfer made from IDIA_ND to MANCHESTER RSE using ESCAPEEs Rucio "production" instance (did: 0_113e8e13-c8c1-4916-8516-66edf3fc3649) on 16/05/22. This was done via the rucio-analysis framework.

      Some complications arose due to restricted permissions when creating new subscriptions, so had to switch from userpass access to x509. A review of the process can be found here

      Show
      Successful subscription based transfer made from IDIA_ND to MANCHESTER RSE using ESCAPEEs Rucio "production" instance (did: 0_113e8e13-c8c1-4916-8516-66edf3fc3649) on 16/05/22. This was done via the rucio-analysis framework. Some complications arose due to restricted permissions when creating new subscriptions, so had to switch from userpass access to x509. A review of the process can be found  here
    • 14.6
    • Stories Completed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI24 - UNCOVERED

    Description

      Who?

      Developers and architects working on SRCs, SDP, and shared services.

      Longer-term: Eventually AA Operators.

      What?

      Support team NALEDI in getting acquainted with existing SKAO Rucio testing infrastructure. This includes:

      • Ensuring that the existing production testbed is in working order (monitoring, transfers etc.)
      • Knowledge sharing of the rucio-analysis (https://github.com/ESCAPE-WP2/rucio-analysis) package and demonstrating how it can be used to instantiate and schedule transfer tests between sites in the datalake.
      • Some support should problems be encountered.

       

      Why?

      Aim to understand the potential use of Rucio as a solution for the early Array Assemblies, starting as soon as AA0.5.

      Rucio is a leading solution for managing large amounts of data, capable of managing large volumes of data and a large number of files/objects, and is a mature community-supported solution that meets the current understanding of SKA and SRC use cases. 

      Other notes:

      • Anticipated/expected to be a collaboration with SP-1547

       

      Attachments

        Issue Links

          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-1547 SDP Data delivery

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                j.coles Coles, Jeremy
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete35.0
                  Total35.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel