Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2832

Integrate Rucio with SRC IAM for auth provision

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The Rucio instance has been integrated with the ESCAPE IAM service historically. Since the SRC IAM service is now up and running and ESCAPE is drawing to a close, it would be good to migrate to using the SRC IAM in order to prevent accruing of further technical debt. If more storage sites are added to the datalake before this migration is done, we would be adding to the collective technical debt across the ART.

      Update: This is becoming especially critical now we're working on APIs. At the moment the data-management IAM client is managed by the ESCAPE IAM instance as some endpoints require token exchanges against the Rucio "auth" client (which has always lived on the ESCAPE IAM instance) to do work within the Rucio ecosystem. As such, the data-management API cannot be integrated with any other API (e.g. permissions) with clients utilising the SKA IAM.

      Show
      The Rucio instance has been integrated with the ESCAPE IAM service historically. Since the SRC IAM service is now up and running and ESCAPE is drawing to a close, it would be good to migrate to using the SRC IAM in order to prevent accruing of further technical debt. If more storage sites are added to the datalake before this migration is done, we would be adding to the collective technical debt across the ART. Update: This is becoming especially critical now we're working on APIs. At the moment the data-management IAM client is managed by the ESCAPE IAM instance as some endpoints require token exchanges against the Rucio "auth" client (which has always lived on the ESCAPE IAM instance) to do work within the Rucio ecosystem. As such, the data-management API cannot be integrated with any other API (e.g. permissions) with clients utilising the SKA IAM.
    • Hide

      AC: Rucio functional tests running (at all deployed sites) with an SRC IAM use, verified via the Grafana dashboard, and token based CLI flow verified for a user registered with SRC IAM

      AC: Rucio functional tests running (at all deployed sites) with an SRC FTS use, verified via the Grafana dashboard

      Show
      AC: Rucio functional tests running (at all deployed sites) with an SRC IAM use, verified via the Grafana dashboard, and token based CLI flow verified for a user registered with SRC IAM AC: Rucio functional tests running (at all deployed sites) with an SRC FTS use, verified via the Grafana dashboard
    • 0.5
    • 1
    • 0
    • Team_MAGENTA
    • Sprint 5
    • Hide

      After much discussion with the FO and PT, we have descoped this feature. Recommended by Rob Perry that we clone the feature to capture remaining work (https://jira.skatelescope.org/browse/SP-4591) and modify this one to capture the switch over to SRC FTS.

      Since Weds 14 Aug, the tests have been running: https://monit.srcdev.skao.int/grafana/d/BfW0IQQVk/rucio-events?orgId=1&from=1723590000000&to=now&var-src=All&var-dst=All&var-protocol=All&var-scopes=testing&var-filename=%2A&var-bin=$__auto_interval_bin&var-throughput_min_filesize_bytes=1000000. 

      Show
      After much discussion with the FO and PT, we have descoped this feature. Recommended by Rob Perry that we clone the feature to capture remaining work ( https://jira.skatelescope.org/browse/SP-4591 ) and modify this one to capture the switch over to SRC FTS. Since Weds 14 Aug, the tests have been running: https://monit.srcdev.skao.int/grafana/d/BfW0IQQVk/rucio-events?orgId=1&from=1723590000000&to=now&var-src=All&var-dst=All&var-protocol=All&var-scopes=testing&var-filename=%2A&var-bin=$__auto_interval_bin&var-throughput_min_filesize_bytes=1000000.  
    • 24.3
    • Stories Completed, Integrated, Outcomes Reviewed
    • PI24 - UNCOVERED

    • SRC-AAI SRC-DM SRC-Multi-Team SRC23-PB SRCNet0.1 operations-and-infrastructure

    Description

      This will require:

      • Liaising with FTS pilot instance to trust the SRC IAM auth provider (1SP)
        • Note: If we can get the timing of this right, and have our own SRC FTS ready to be integrated in the Rucio ecosystem, we would do away with this external dependency - update 30-05-24: understand that SKA FTS is not ready, so will press ahead on CERN Pilot.
      • Creating new auth/admin clients on the SKA IAM & adjust idpsecrets.json in Rucio secret (0.5SP)
      • Adjusting the sync script in Rucio task manager, remove all existing accounts and resync (2SP)
      • Having all existing sites switch any token provider fields in configuration from ESCAPE to IAM (1SP to communicate; small one line change and service restart required by each site - should only take ~10 mins but needs RSE operator to be available)

      Descoped after discussing with FO, PM and Project Lead. Prudent to build trust in SRC FTS for a few more weeks, and schedule the main server switch to SRC IAM when operators are more available (after August) - in the start of PI25.

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                J.Walder Walder, James
                M.Parra Parra, Manuel
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete44.5
                  Total44.5

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel