Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3680

Build and deploy a data ingestion service to complement the current data management API

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • SRCnet
    • Hide

      The current implementation of the data-management API performs a token exchange that adds a claim to a user's token allowing them to write to a (restricted) piece of storage. The other side of this process is a service that will subsequently ingest the data into the datalake when it has been written to the storage. This is key functionality both in the SDP interface (prospectively...) and user space (for ADPs).

      This would complement the current DM API ecosystem and allow for creation of new replicas as described in SRCNet v0.1, it would also feed into building a staging area for users (v0.2).

      Show
      The current implementation of the data-management API performs a token exchange that adds a claim to a user's token allowing them to write to a (restricted) piece of storage. The other side of this process is a service that will subsequently ingest the data into the datalake when it has been written to the storage. This is key functionality both in the SDP interface (prospectively...) and user space (for ADPs). This would complement the current DM API ecosystem and allow for creation of new replicas as described in SRCNet v0.1, it would also feed into building a staging area for users (v0.2).
    • Hide

      AC: Solution implemented that can ingest data sitting on a piece of storage into the datalake

      AC: This solution deployed as part of the DM API

      AC: Upload demonstrated.

      Show
      AC: Solution implemented that can ingest data sitting on a piece of storage into the datalake AC: This solution deployed as part of the DM API AC: Upload demonstrated.
    • 1
    • 1
    • 0
    • Team_MAGENTA
    • Sprint 5
    • Hide

      Code and docs (plus demo video): https://gitlab.com/ska-telescope/src/ska-src-ingestion 

      System demo: https://confluence.skatelescope.org/display/SRCSC/2023-11-16+SRC+ART+System+Demo+20.5+Part+2+AM 

      Show
      Code and docs (plus demo video): https://gitlab.com/ska-telescope/src/ska-src-ingestion   System demo: https://confluence.skatelescope.org/display/SRCSC/2023-11-16+SRC+ART+System+Demo+20.5+Part+2+AM  
    • 20.6
    • Stories Completed, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI24 - UNCOVERED

    • PI20-PB

    Description

      Finish adding upload to datalake functionality

      Attachments

        Issue Links

          relates to

          Feature - A Feature is a service that fulfills a stakeholder need. Each feature includes a benefit hypothesis and acceptance criteria, and is sized or split as necessary to be delivered by a single Agile Release Train (ART) in a Program Increment (PI). SP-4227 Define requirements for SRCNet Ingestion Nodes

          • Must have - Items labelled as "Must have" are critical to the current delivery timebox in order for it to be a success.
          • Done
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Structure

            Activity

              People

                r.bolton Bolton, Rosie
                r.joshi Joshi, Rohini
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete36.0
                  Total36.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel