Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3671

Testing FTS and configuring XrootD

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Not Assigned
    • PI20
    • None
    • National SRC
    • Hide

      The UKSRC will transfer data between sites, and needs to build and configure xrootd with security & tokens at Cambridge HPC, GEANT, JBO and STFC RAL.

      Establishing and demonstrating efficient and secure transfers between sites is a critical task of SRCNet and for the UKSRC therefore this is the necessary next step in developing the data transfer stack for UKSRC. It is therefore a national priority, but is necessary to allow future work which will be of relevance more widely.

      The creation of a testbed of a site mesh will establish the capabilities to transfer data into, out of, and within the UK using standard sets of software. Once demonstrated, it will enable further end-to-end transfer tests, including at scale, the capabilities to test new types of SEs, file types, file sizes , etc.

      The work proposed here underpins and creates the foundation for further studies and demonstrators for data management, AAI, and to facilitate the onboarding of communities new to the volumes of data to the tooling that is likely to be used. With the token based developments sufficiently mature and a demand for these tests to be started, this PI is well-placed to realise the needed infrastructure.

      This should be considered as an International Priority as the knowledge gained will be vital to feedback into SRCNet.

      Show
      The UKSRC will transfer data between sites, and needs to build and configure xrootd with security & tokens at Cambridge HPC, GEANT, JBO and STFC RAL. Establishing and demonstrating efficient and secure transfers between sites is a critical task of SRCNet and for the UKSRC therefore this is the necessary next step in developing the data transfer stack for UKSRC. It is therefore a national priority, but is necessary to allow future work which will be of relevance more widely. The creation of a testbed of a site mesh will establish the capabilities to transfer data into, out of, and within the UK using standard sets of software. Once demonstrated, it will enable further end-to-end transfer tests, including at scale, the capabilities to test new types of SEs, file types, file sizes , etc. The work proposed here underpins and creates the foundation for further studies and demonstrators for data management, AAI, and to facilitate the onboarding of communities new to the volumes of data to the tooling that is likely to be used. With the token based developments sufficiently mature and a demand for these tests to be started, this PI is well-placed to realise the needed infrastructure. This should be considered as an International Priority as the knowledge gained will be vital to feedback into SRCNet.
    • Hide

      AC1: demoing the xrootd endpoints together with their setup/configuration (some material already exists).

      AC2: SRCnet Demo on Data Movement Performance between UKSRC sites using the SRC FTS instance

      AC3: SRCnet Demo on the xrootd endpoints and how they have been setup/configured

      Show
      AC1: demoing the xrootd endpoints together with their setup/configuration (some material already exists). AC2: SRCnet Demo on Data Movement Performance between UKSRC sites using the SRC FTS instance AC3: SRCnet Demo on the xrootd endpoints and how they have been setup/configured
    • Overdue

    • PI24 - UNCOVERED

    • Teal-B UKSRC

    Description

      The UKSRC  build and configure xrootd with security & tokens at Cambridge HPC, GEANT and JBO; it will then measure performance in transferring data between sites.

      Xrootd servers are used in the WLCG community but at lower site to site round trip latencies, lower connection speeds (~25Gb) and with potentially different file size distributions than currently proposed for SRCnet. So there is technical learning in these areas for SRCnet and xrootd communities as well as the token based authentication. The results from this work will feed into the SRCnet NREN forum architecture.

      Definitions:

      xrootd is a distributed, scalable, high-performance, and low-latency software framework that provides seamless access to data through various means, including local file systems, the World Wide Web, and cloud-based storage.

      Security is required so that data transfers can be authenticated, authorised, and encrypted, maintaining the confidentiality and integrity of the data being transferred.

      Tokens is currently the accepted tooling for authorising transfers (VOMS is generally being deprecated).

      FTS is a service that manages and tracks large-scale data transfers across grid computing environments. FTS is a potential solution to the orchestration of managing the transfer of data,  and needs to interact with configured SEs.

      Rucio enables the cataloguing of data, and provides the tooling for managing the data over its lifecycle, creating virtual Rucio Storage Endpoints (RSEs) for each of the associated SEs.

       

      Attachments

        Issue Links

          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3775 Complete finalisation and testing of SRCNet FTS instance

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-3502 Finalise and test deployment of SRCNet FTS instance

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Discarded
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4591 Switch Rucio server to SRC IAM for auth provision

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Releasing

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-2832 Integrate Rucio with SRC IAM for auth provision

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          mentioned in

          Page Loading...

          Structure

            Activity

              People

                r.hughes-jones Richard Hughes-Jones
                L.Chisholm Chisholm, Louise
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel