Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4517

Local DM Service: prepareData function linking operation

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      This feature will provide information on how to implement the approach "moving the code to data", avoiding data transfer and also avoiding creating a copy of the data for a particular user, by creating a link to the data product into the DDM repository

      Show
      This feature will provide information on how to implement the approach "moving the code to data", avoiding data transfer and also avoiding creating a copy of the data for a particular user, by creating a link to the data product into the DDM repository
    • Hide

      AC1: Demo showing how a user can analyse a dataset that is provided using this method. User connect to a JHub session and s/he can see the dataset which is linked to the Rucio DataLake. The A&A part is not covered here, so we will assume that the user has access rights to the data .

      Show
      AC1: Demo showing how a user can analyse a dataset that is provided using this method. User connect to a JHub session and s/he can see the dataset which is linked to the Rucio DataLake. The A&A part is not covered here, so we will assume that the user has access rights to the data .
    • 1
    • 1
    • 0
    • Team_CORAL
    • Sprint 4
    • Show
      Demo: https://confluence.skatelescope.org/pages/viewpage.action?pageId=294227996 Repository: Pre-Alpha version: https://gitlab.com/ska-telescope/src/src-dm/ska-src-dm-local-data-preparer/-/tree/coral-approach-preparedata?ref_type=heads
    • 24.3
    • Demonstrated
    • PI24 - UNCOVERED

    • PI24-PB SRCNet0.1
    • SPO-3479

    Description

      One of the compulsory services at the SRCNet v0.1 nodes is the Local Data Management Service, which will be in charge of prepare the datafile/s that is/are going to be analysed in the storage area attached to the local computing environment
      This service will get as input:

      • the ID (or IDs) of a specific datafile/s stored in the Rucio Data Lake
      • the user token

      with this input, the service will link the corresponding datafile/s into the scratch area attached to the computing environment that will be used to analyze it/them.

      In particular the local DM will:

      • find out the corresponding local user (from the user token)
      • create a symbolic link in the user area attached to the computing environment. 

      We can use CANFAR to do this implementation in this way:

      • Assuming CANFAR PosixMapper maps the "network users" (whose userID can be get from the user token) to the local user ( same network user will always be mapped to the same local user), we can get the local path (at operating system level) of the user storage area.
      • Assuming CANFAR mounts the Local RSE in read-only mode (this has been recently demoed by M. Parra), we can create a symbolic link (at operating system level) from the data file in the Local RSE volume to the local user storage area

       

      This service will implement calls in line with the to be agreed interface defined at:

      https://jira.skatelescope.org/browse/SP-4678

       

      Attachments

        Issue Links

          Structure

            Activity

              People

                Jesus.Salgado Salgado, Jesus
                S.SanchezExposito Sanchez Exposito, Susana
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete38.0
                  Total38.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel