Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3543

Explore S3-S3 Rucio transfers via streaming transfers through FTS

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Not Assigned
    • PI20
    • None
    • SRCnet
    • Hide

      Exploring Rucio support for S3 storage has been ongoing over the last several PIs and we have a (Swiss) S3 endpoint added to the datalake that supports uploads. However, until we look at how data transfers can be done in/out of this endpoint, it will remain a bit of an island. Since there is additional knowledge and documentation since deploying the Swiss S3 endpoint, this feature will entail another S3 endpoint and explore s3-s3 streamed transfers to/from the existing S3 endpoint and a newly added one.

      Show
      Exploring Rucio support for S3 storage has been ongoing over the last several PIs and we have a (Swiss) S3 endpoint added to the datalake that supports uploads. However, until we look at how data transfers can be done in/out of this endpoint, it will remain a bit of an island. Since there is additional knowledge and documentation since deploying the Swiss S3 endpoint, this feature will entail another S3 endpoint and explore s3-s3 streamed transfers to/from the existing S3 endpoint and a newly added one.
    • Hide

      Demonstrate S3-S3 streaming transfers within FTS/Rucio OR document outlining hurdles in doing the same, and if applicable development work needed in the software stack involved (gfal, Rucio, FTS, etc) is identified.

      Show
      Demonstrate S3-S3 streaming transfers within FTS/Rucio OR document outlining hurdles in doing the same, and if applicable development work needed in the software stack involved (gfal, Rucio, FTS, etc) is identified.
    • 24.2

    • PI24 - UNCOVERED

    • PI19-PB PI20-PB

    Description

      Add another S3 endpoint to Rucio datalake, to enable exploring S3-S3 transfers. 

      Probably easier when we have our own SRCNet FTS to limit async communication

      Document outcomes/hurdles.

      Have been informed by FTS Project lead that this is possible only by streaming the transfer via FTS, and not TPC. 

      Attachments

        Issue Links

          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3775 Complete finalisation and testing of SRCNet FTS instance

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3502 Finalise and test deployment of SRCNet FTS instance

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Discarded
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3542 Perform TPC and/or streamed transfers to/from an S3 storage endpoint in the Rucio data lake

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel