Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2903

Datalake access from containerised services

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      We will save the users the work to authenticate in the data lake and retrieve the data that they need. In the process of starting up the container, the user will be authenticated in the data lake and the data that they need will be retreived to the RSE next to where the container is running. The users will have access to a container ready to work.

      The data retrieved will be located in the RSE, so if in the future a collaborator or the same user needs to work with the same dataset, it will not be needed to transfer it again.

       

      Show
      We will save the users the work to authenticate in the data lake and retrieve the data that they need. In the process of starting up the container, the user will be authenticated in the data lake and the data that they need will be retreived to the RSE next to where the container is running. The users will have access to a container ready to work. The data retrieved will be located in the RSE, so if in the future a collaborator or the same user needs to work with the same dataset, it will not be needed to transfer it again.  
    • Hide

      AC: A special container will be created to be able to accept as environment variables the username and the file/s that the user requires from the data lake, and to authenticate the user and retrieve the file/s

      AC2: In case an orchestrator such as kubernetes is in charge of managing the container execution, this orchestrator should be able to launch the container with the corresponding environment variables.

      Show
      AC: A special container will be created to be able to accept as environment variables the username and the file/s that the user requires from the data lake, and to authenticate the user and retrieve the file/s AC2: In case an orchestrator such as kubernetes is in charge of managing the container execution, this orchestrator should be able to launch the container with the corresponding environment variables.
    • 22.6

    • PI23 - UNCOVERED

    • SRC-DM SRC-DataAccess SRC-SWdist SRCNet0.1 Team_CORAL data-lifecycle

    Description

      In the same way we needed to integrate as seamlessly as possible JHub and Rucio, we would like to do with other applications. A general case is when a user access to a container (which may have installed the tools required for the data analysis) to analyse a set of data stored in the data lake. We would like to avoid the manual steps required for the autenticantion (token capture) and the download of the dataset/data required by the user.

      The idea is to provide the user with a container that would be responsible for authenticating the user as automatically as possible, download the data and make them available to the user. A way to do this is to launch the container using environmental variables:  docker run .... -e user=mparra  python test.py <datasetdid>
      Then the container would take care of the above process.

      This is related to https://jira.skatelescope.org/browse/SP-2654?src=confmacro and https://jira.skatelescope.org/browse/SP-2902

      Who: Coral team with support from Cyan Team

      What: Investigate how to automatize/facilitate the accesss to data in the data lake from containers (see link)

      Why: Enable availability of access to Datalake from applications or services that are containerised.

      Attachments

        Issue Links

          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                j.collinson Collinson, James
                j.collinson Collinson, James
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel