Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4391

Demonstrate CNSRC CANFAR reading SRC Data Lake data in-place on the CNSRC Rucio Storage Element

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Team Magenta has previously demonstrated that a CANFAR science platform deployment running in the same location as a Rucio storage element (RSE) can read data directly from the RSE using volume mounts. This means that user sessions spawned by the CANFAR service can process/visualise the data lake data without having to move it.

      Replicating this at CNSRC will be a good verification exercise of both the RSE and CANFAR instances running in CNSRC, as well as building experience within the Gold Team.

      Show
      Team Magenta has previously demonstrated that a CANFAR science platform deployment running in the same location as a Rucio storage element (RSE) can read data directly from the RSE using volume mounts. This means that user sessions spawned by the CANFAR service can process/visualise the data lake data without having to move it. Replicating this at CNSRC will be a good verification exercise of both the RSE and CANFAR instances running in CNSRC, as well as building experience within the Gold Team.
    • Hide

      AC1: CNSRC has both a stable RSE (with >80% of functional test transfers passing according to dashboard) and publicly accessible CANFAR science platform (accessible to all SRCNet developers with an SKA IAM account).

      AC2: Demo of a user session spawned by the CNSRC CANFAR Science Platform reading data directly from the RSE.

      AC3: Demo of one of the science workflow tasks from the SRC Workloads repository (https://gitlab.com/ska-telescope/src/src-workloads) running on the CNSRC CANFAR Science Platform, reading data directly in-place.

      Stretch Goal:

      Add and test Local IAM to CANFAR Platform.

      Add Harbor to k8s in CNSRC.

      Show
      AC1 : CNSRC has both a stable RSE (with >80% of functional test transfers passing according to dashboard) and publicly accessible CANFAR science platform (accessible to all SRCNet developers with an SKA IAM account). AC2 : Demo of a user session spawned by the CNSRC CANFAR Science Platform reading data directly from the RSE. AC3 : Demo of one of the science workflow tasks from the SRC Workloads repository ( https://gitlab.com/ska-telescope/src/src-workloads ) running on the CNSRC CANFAR Science Platform, reading data directly in-place. Stretch Goal: Add and test Local IAM to CANFAR Platform. Add Harbor to k8s in CNSRC.
    • 1.5
    • 1.5
    • 0
    • Team_GOLD

    • PI23 - UNCOVERED

    • SRC23-PB SRCNet0.1 science-platform-services

    Description

      Work required will depend on how many prerequisites are met. Assuming there is already a CANFAR instance and RSE running at CNSRC, this ought not require much additional work. The notes at: https://confluence.skatelescope.org/display/SRCSC/SP-3729+CANFAR+Science+Platform+with+RSE+Storage+attached will hopefully help.

      • Make RSE storage available to CANFAR host K8s cluster worker nodes (this can be done with NFS or similar)
      • Modify the CANFAR Science Platform Job configurations which launch the sessions (will need to modify a Job for each session type to be modified - notebook, carta, desktop, etc)
      • Re-run one of the src-workloads tasks reading the data in-place. To discover the local data file paths, the astroquery module can be used, or these can be hardcoded for the purposes of a demo.

      Attachments

        Structure

          Activity

            People

              j.collinson Collinson, James
              Robert.Perry Perry, Robert
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Feature Progress

                Story Point Burn-up: (0%)

                Feature Estimate: 1.5

                IssuesStory Points
                To Do816.0
                In Progress   23.0
                Complete00.0
                Total1019.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel