Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3667

Review and consolidate design/architecture for data management

    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      It is becoming increasingly urgent that SKA has a consolidated and mature data management architecture and strategy considering short-term (AA0.5) and longer-term (AA1+) needs.

      While we have a working starting point for this, some of which are already being implemented for AA0.5 (ADR-55, ADR-80, ADR-82), we believe PI20 is a good time/opportunity to review the approach, develop alternatives and consolidate these against current plans.

      Show
      It is becoming increasingly urgent that SKA has a consolidated and mature data management architecture and strategy considering short-term (AA0.5) and longer-term (AA1+) needs. While we have a working starting point for this, some of which are already being implemented for AA0.5 ( ADR-55 , ADR-80 , ADR-82 ), we believe PI20 is a good time/opportunity to review the approach, develop alternatives and consolidate these against current plans.
    • Hide
      1. The Solution Intent confluence space includes a landing section for data management, highlighting the main decisions and principles related to SKA implementation and pointing to relevant ADRs and highlighting the main non functional requirements for all SKA data products. 
      2. ADRs are reviewed, consolidated, and alternative solutions proposed producing a consolidated picture for data management architecture in the Solution Intent considering:
        • AA0.5
        • AA1+ (with a focus more on AA2 when data volumes increase considerably in the form of observatory data products)
      3. For the data products we are working on, a way to describe them in detail as part of the Solution Intent is identified. This can include an initial conversation about the concept related to ownership of data products and definition of data product owners and it will need to involve the OSO teams and the science operations teams to start with. 
      4. A draft roadmap that is ready for review/discussion considering steps that need to be taken for the solution produced by item 1.
      5. Backlog proposals generated for follow on work in the next 2-3 PIs.
      Show
      The Solution Intent confluence space includes a landing section for data management, highlighting the main decisions and principles related to SKA implementation and pointing to relevant ADRs and highlighting the main non functional requirements for all SKA data products.  ADRs are reviewed, consolidated, and alternative solutions proposed producing a consolidated picture for data management architecture in the Solution Intent considering: AA0.5 AA1+ (with a focus more on AA2 when data volumes increase considerably in the form of observatory data products) For the data products we are working on, a way to describe them in detail as part of the Solution Intent is identified. This can include an initial conversation about the concept related to ownership of data products and definition of data product owners and it will need to involve the OSO teams and the science operations teams to start with.  A draft roadmap that is ready for review/discussion considering steps that need to be taken for the solution produced by item 1. Backlog proposals generated for follow on work in the next 2-3 PIs.
    • Inter Program
    • 2
    • 2
    • 0
    • Team_YANDA
    • Sprint 3
    • Hide

      The outcome of this work is presented in a number of confluence pages - these pages address the AC in the following manner:

      AC 1 The Solution Intent confluence space includes a landing section for data management, highlighting the main decisions and principles related to SKA implementation and pointing to relevant ADRs and highlighting the main non functional requirements for all SKA data products.

      This is met by the pages in confluence Data Management (ska-data)
      AC 2 ADRs are reviewed, consolidated, and alternative solutions proposed producing a consolidated picture for data management architecture in the Solution Intent considering:

      • AA0.5
      • AA1+ (with a focus more on AA2 when data volumes increase considerably in the form of observatory data products)

      The response to the ADRs is listed in the "Related Activities":  The review of the ADRs is not explicit in the pages, but the model has been adjusted to reflect the current implementation and thus the ADRs.
      AC 3 For the data products we are working on, a way to describe them in detail as part of the Solution Intent is identified. This can include an initial conversation about the concept related to ownership of data products and definition of data product owners and it will need to involve the OSO teams and the science operations teams to start with.

      There are sections concerning Policies and Responsibilties and the data product descriptions
      AC 4 A draft roadmap that is ready for review/discussion considering steps that need to be taken for the solution produced by item 1.

      DLM Roadmap section is present
      AC 5 Backlog proposals generated for follow on work in the next 2-3 PIs.

      A continuing feature is in the funnel

       https://jira.skatelescope.org/browse/SP-3860

      Show
      The outcome of this work is presented in a number of confluence pages - these pages address the AC in the following manner: AC 1 The Solution Intent confluence space includes a landing section for data management, highlighting the main decisions and principles related to SKA implementation and pointing to relevant ADRs and highlighting the main non functional requirements for all SKA data products. This is met by the pages in confluence Data Management (ska-data) AC 2 ADRs are reviewed, consolidated, and alternative solutions proposed producing a consolidated picture for data management architecture in the Solution Intent considering: AA0.5 AA1+ (with a focus more on AA2 when data volumes increase considerably in the form of observatory data products) The response to the ADRs is listed in the "Related Activities":  The review of the ADRs is not explicit in the pages, but the model has been adjusted to reflect the current implementation and thus the ADRs. AC 3 For the data products we are working on, a way to describe them in detail as part of the Solution Intent is identified. This can include an initial conversation about the concept related to ownership of data products and definition of data product owners and it will need to involve the OSO teams and the science operations teams to start with. There are sections concerning Policies and Responsibilties and the data product descriptions AC 4 A draft roadmap that is ready for review/discussion considering steps that need to be taken for the solution produced by item 1. DLM Roadmap section is present AC 5 Backlog proposals generated for follow on work in the next 2-3 PIs. A continuing feature is in the funnel   https://jira.skatelescope.org/browse/SP-3860
    • 20.6
    • Stories Completed, Solution Intent Updated, Outcomes Reviewed, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO

    • PI22 - UNCOVERED

    • Team_YANDA

    Description

      Review and consolidate alternatives for design/architecture for data management considering immediate (AA0.5 & possibly AAVS3) and longer term (AA1+) impact

      Outcome: ADRs, Solution Intent pages & architectural/technology roadmap for SKA data management up to ~AA2 Timeboxed to ~1-2FP and needs to happen early in the PI with a view that more enablers/features created following this later in the PI as needed

      Input ADRs: ADR-55, ADR-80, ADR-82

      Attachments

        Issue Links

          informs

          Spike - A time limited investigation to get some knowledge SP-3811 Consolidate ad-hoc data management efforts

          • Not Assigned - Priority is not assigned yet.
          • Funnel

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-3809 Initial implementation of data management

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Releasing
          is cloned by

          Enabler - An Enabler supports the activities needed to extend the Architectural Runway to provide future business functionality. These include exploration, architecture, infrastructure, and compliance. SP-3860 Data Management Use Case Analysis

          • Should have - Items labelled as "Should have" are important but not necessary for delivery in the current delivery timebox. While "Should have" items can be as important as "Must have", they are often not as time-critical or there may be another way to satisfy the requirement so that it can be held back until a future delivery timebox.
          • Funnel
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Wiki Page

          Wiki Page Loading...

          Structure

            Activity

              People

                m.bartolini Bartolini, Marco
                b.mort Mort, Ben
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete612.0
                  Total612.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel