Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4162

Data management API extended to include data movement capability

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      Having this implemented would allow data to be moved after ingestion, this is useful for e.g. moving data to local nodes for use in interactive environments

      Show
      Having this implemented would allow data to be moved after ingestion, this is useful for e.g. moving data to local nodes for use in interactive environments
    • Hide

      AC: The endpoints made and data transfer between two sites demonstrated using these endpoints.

      Show
      AC: The endpoints made and data transfer between two sites demonstrated using these endpoints.
    • 1
    • 1
    • 0
    • Team_MAGENTA
    • Hide

      R.Barnsley added a comment
       
      I've mentioned this in https://jira.skatelescope.org/browse/SP-4101 but to reiterate, this is done: https://drive.google.com/file/d/1Ka8RS07nYwBSYwCRs5x6pTmo47xjvBL3/view?usp=sharing

      There are now endpoints to:

      a) "request movement", which, in the context of Rucio, attaches all the data requested to a parent container and creates a rule (with unique ID, which is used as the "job id" in the nomenclature defined above) to transfer this to the requested site, and 
      b) get the status of this movement via polling

      Show
      R.Barnsley added a comment   I've mentioned this in  https://jira.skatelescope.org/browse/SP-4101  but to reiterate, this is done:  https://drive.google.com/file/d/1Ka8RS07nYwBSYwCRs5x6pTmo47xjvBL3/view?usp=sharing There are now endpoints to: a) "request movement", which, in the context of Rucio, attaches all the data requested to a parent container and creates a rule (with unique ID, which is used as the "job id" in the nomenclature defined above) to transfer this to the requested site, and  b) get the status of this movement via polling
    • 22.6
    • Outcomes Reviewed

    • PI23 - UNCOVERED

    • SRCNet0.1 data-ingestion-dissemination-and-replication data-lifecycle

    Description

      Within the DM API, data placement at sites can currently only be specified at ingestion. This feature is to implement new DM endpoints that take a list of data identifiers and move (if required) the data from one site to another.

      In the context of Rucio, this could be two endpoints that, e.g.

      1. Asynchronously creates the necessary "rules", creates a reference to the ruleset by a persisted job id and,
      2. Checks a job id's rules have been fulfilled and data has been moved.

      Attachments

        Issue Links

          is required by

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4101 Define Science Gateway - Data Management Server Interface for selected data

          • Could have - Items labelled as "Could have" are desirable but not necessary and could improve the user experience or customer satisfaction for a little development cost. These will typically be included if time and resources permit.
          • Implementing
          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4421 Stage 1 System Architecture Design for Distributed Compute and Storage

          • Not Assigned - Priority is not assigned yet.
          • Funnel
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.bolton Bolton, Rosie
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (0%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete00.0
                  Total00.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel