Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4591

Switch Rucio server to SRC IAM for auth provision

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The Rucio instance has been integrated with the ESCAPE IAM service historically. Since the SRC IAM service is now up and running and ESCAPE is drawing to a close, it would be good to migrate to using the SRC IAM in order to prevent accruing of further technical debt. If more storage sites are added to the datalake before this migration is done, we would be adding to the collective technical debt across the ART.

      Update: This is becoming especially critical now we're working on APIs. At the moment the data-management IAM client is managed by the ESCAPE IAM instance as some endpoints require token exchanges against the Rucio "auth" client (which has always lived on the ESCAPE IAM instance) to do work within the Rucio ecosystem. As such, the data-management API cannot be integrated with any other API (e.g. permissions) with clients utilising the SKA IAM.

      In PI23, a number of blockers for this work were identified, and then worked around. Chief among these was the inability to continue to use the CERN Pilot FTS for SRCNet 0.1. This meant that we had to switch to using the SRCNet FTS instance instead, initially with ESCAPE IAM tokens. Given that this only occurred in Sprint 5, we felt it sensible to allow some time for the functional tests to verify the reliability of this FTS instance, and give the RSE operators some advance warning of the impending switch to SKA IAM (some small config changes are required to the RSE storage resource managers to enable this)

      Show
      The Rucio instance has been integrated with the ESCAPE IAM service historically. Since the SRC IAM service is now up and running and ESCAPE is drawing to a close, it would be good to migrate to using the SRC IAM in order to prevent accruing of further technical debt. If more storage sites are added to the datalake before this migration is done, we would be adding to the collective technical debt across the ART. Update: This is becoming especially critical now we're working on APIs. At the moment the data-management IAM client is managed by the ESCAPE IAM instance as some endpoints require token exchanges against the Rucio "auth" client (which has always lived on the ESCAPE IAM instance) to do work within the Rucio ecosystem. As such, the data-management API cannot be integrated with any other API (e.g. permissions) with clients utilising the SKA IAM. In PI23, a number of blockers for this work were identified, and then worked around. Chief among these was the inability to continue to use the CERN Pilot FTS for SRCNet 0.1. This meant that we had to switch to using the SRCNet FTS instance instead, initially with ESCAPE IAM tokens. Given that this only occurred in Sprint 5, we felt it sensible to allow some time for the functional tests to verify the reliability of this FTS instance, and give the RSE operators some advance warning of the impending switch to SKA IAM (some small config changes are required to the RSE storage resource managers to enable this)
    • Hide

      AC: Rucio functional tests running (at all deployed sites) with an SRC IAM use, verified via the Grafana dashboard, and token based CLI flow verified for a user registered with SRC IAM

       

      Show
      AC: Rucio functional tests running (at all deployed sites) with an SRC IAM use, verified via the Grafana dashboard, and token based CLI flow verified for a user registered with SRC IAM  
    • 0.5
    • 1
    • 0
    • Team_MAGENTA
    • Sprint 1
    • Hide

      Can be tested by authing with the rucio client. 

      General status:

      https://monit.srcdev.skao.int/grafana/public-dashboards/2846c870a07d48c68a046886918cbac6?orgId=1

      Transfers following switchover (dotted line)

       

      Show
      Can be tested by authing with the rucio client.  General status: https://monit.srcdev.skao.int/grafana/public-dashboards/2846c870a07d48c68a046886918cbac6?orgId=1 Transfers following switchover (dotted line)  
    • PI24 - UNCOVERED

    • PI24-PB SRC-AAI SRC-DM SRC-Multi-Team SRC23-PB SRCNet0.1 operations-and-infrastructure
    • SPO-3479

    Description

      This will require:

      • Modify Rucio server config to point to SRCNet IAM (1SP)
      • Adjusting the sync script in Rucio task manager, remove all existing accounts and resync (2SP)
      • Having all existing sites switch any token provider fields in configuration from ESCAPE to IAM (1SP to communicate; small one line change and service restart required by each site - should only take ~10 mins but needs RSE operator to be available)

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                J.Walder Walder, James
                M.Parra Parra, Manuel
                Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 0.5

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete66.0
                  Total66.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel