Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3809

Initial implementation of data management

    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      Provide fundamental service to enable the ability to ingest, track, trace, find and access data products of any kind within the SKAO. This will be using a generic data management schema initially, but allow for product specific schemata as well.

      Show
      Provide fundamental service to enable the ability to ingest, track, trace, find and access data products of any kind within the SKAO. This will be using a generic data management schema initially, but allow for product specific schemata as well.
    • Hide
      1. Introduce service that can be used to track for intermediate and final data products by ID (as of ADR-54):
        1. data product metadata (initially as of ADR-54)
        2. Location information (should allow a number of storage backends, including informal ones like something that is in storage on an HPC system like CSD3 or Pawsay)
        3. Lifecycle information (at minimum lifetime)
        4. Updateable user annotations?
      2. Integrate into SDP
        1. Stretch - have data product dashboard show information from it
        2. Stretch - consider how this could be ported/synchronised to enable a global SRC infrastructure
        3. Stretch - consider authentication (this would need to go via execution block ID metadata)
      Show
      Introduce service that can be used to track for intermediate and final data products by ID (as of ADR-54 ): data product metadata (initially as of ADR-54 ) Location information (should allow a number of storage backends, including informal ones like something that is in storage on an HPC system like CSD3 or Pawsay) Lifecycle information (at minimum lifetime) Updateable user annotations? Integrate into SDP Stretch - have data product dashboard show information from it Stretch - consider how this could be ported/synchronised to enable a global SRC infrastructure Stretch - consider authentication (this would need to go via execution block ID metadata)
    • Inter Program, Intra Program
    • 5
    • 5
    • 0

    • Enabler SP-3667 Review and consolidate design/architecture for data management

    • Team_YANDA
    • Sprint 5
    • Hide

      The implemented services can be used to track for intermediate and final data products by UID, OID and the semantic ID (as of ADR-54):

      • Location information allows multiple storage backends, including informal ones like something that is in storage on local storage and Acacia Pawsay)
      • Lifecycle information both for the whole set of related products, replicas and individual items.
      • Updateable user annotations as JSON keyword/value pairs.
      • Integrated into SDP with make process, helm charts, testing and readthedocs.
      • data product dashboard has been deferred due to reduce availability of Naledi team during this PI.
      • Consideration how this could be ported/synchronised to enable data movements across a global SRC infrastructure.
      • AAA is currently a stub and will be started next PI.
      Show
      The implemented services can be used to track for intermediate and final data products by UID, OID and the semantic ID (as of ADR-54 ): Location information allows multiple storage backends, including informal ones like something that is in storage on local storage and Acacia Pawsay) Lifecycle information both for the whole set of related products, replicas and individual items. Updateable user annotations as JSON keyword/value pairs. Integrated into SDP with make process, helm charts, testing and readthedocs. data product dashboard has been deferred due to reduce availability of Naledi team during this PI. Consideration how this could be ported/synchronised to enable data movements across a global SRC infrastructure. AAA is currently a stub and will be started next PI.

    • PI22 - UNCOVERED

    • Low G3 Mid G3

    Description

      See feature breakdown on PI21 backlog board.

      Old description:

      Following SP-3667 we need to make a start on the implementation of a service underlying the SKA Data Management.

      This acknowledges that at this point in time we can not really make final decisions on:

      1. Responsibilities (setup, design, operations)
      2. Technology choice and installation (e.g. Postgres)
      3. DB schema design

      Therefore we should first focus on building a service-like layer, with a proof-of-concept implementation behind it.

      Links and References

      Attachments

        Issue Links

          is informed by

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-3667 Review and consolidate design/architecture for data management

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Done

          Structure

            Activity

              People

                m.bartolini Bartolini, Marco
                a.wicenec Andreas Wicenec
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 5.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete2033.0
                  Total2033.0

                  Dates

                    Created:
                    Updated:

                    Structure Helper Panel