Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-2834

Improve Rucio uploads with DLaaS

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Feature
    • Won't have (this time)
    • None
    • None
    • SRCnet
    • 17.6

    • PI23 - UNCOVERED

    Description

      Context: The datalake-as-a-service (DLaaS) is a prototype which integrates storage that forms part of a Rucio datalake prototype with JupyterHub, providing a user interface to schedule the staging of data within a user's Jupyter Notebook environment. It was initially developed at CERN within the ESCAPE project, but has since been replicated as part of the SRC prototyping activities (https://confluence.skatelescope.org/display/SRCSC/SKAO+Data+Lake+as+a+Service+Deployment+-+Setup+Guide).

      Summary: Add the ability to upload new data to the datalake from the Rucio-JupyterLab extension.

      Currently, the UI elements for interacting with Rucio from JupyterHub are in the Rucio jupyterlab-extension. When this was first developed (~1 year ago) the token-based upload functionality was lacking in Rucio, so a complex workaround using non-deterministic RSEs was developed. This is no longer necessary as token-based uploads now work, so it should be possible to simplify the jupyterlab-extension to use the existing Rucio upload method.

      AC: UI element in Rucio jupyterlab-extension can be used to upload data into the data lake, at an RSE of the user's choosing.

      Attachments

        Issue Links

          depends on

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-2902 Integrate JupyterHub with the SRCNet Rucio datalake

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Done

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-2654 Review ESCAPE JHub + Rucio integration

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Discarded
          relates to

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-2654 Review ESCAPE JHub + Rucio integration

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Discarded
          mentioned in

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                j.collinson Collinson, James
                j.collinson Collinson, James
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 0.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel