Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-4524

Implement SDP buffer management

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Data Processing
    • Hide

      As a Telescope Operator trying to run many observations in sequence, I want to be sure that full-resolution visibilities are reliably evacuated from the buffer, while valuable data products are kept / migrated to stores managed by data lifecycle management.

      Show
      As a Telescope Operator trying to run many observations in sequence, I want to be sure that full-resolution visibilities are reliably evacuated from the buffer, while valuable data products are kept / migrated to stores managed by data lifecycle management.
    • Hide
      1. Implement data product data flows and associated reservations in the configuration database
        1. Processing scripts should create them and claim reservations immediately
        2. Buffer manager should also create them for "orphaned" data products in the buffer (to represent in the configuration database that these objects are about to be deleted!)
      2. Data lifecycle management should reserve data products that it wants to deliver - might require implementing a data lifecycle management interface service
        1. (Might want to move data product registration model to also be data-flow-based)
        2. Stretch: Implement request to data lifecycle management service to copy/move-in data products
      3. Implement either as part of an SDP buffer manager or platform interface ("Helm deployer") deleting objects without reservations
        1. For a first version we might especially implement it as a move into a "recycle bin" area instead of an actual "delete" - to make sure we don't lose data
        2. Might want to implement some slack (~10 minutes?) to make sure we give other services a chance to add reservations
        3. Could especially disable this entirely if - e.g. - the data lifecycle management interface is not currently active
      Show
      Implement data product data flows and associated reservations in the configuration database Processing scripts should create them and claim reservations immediately Buffer manager should also create them for "orphaned" data products in the buffer (to represent in the configuration database that these objects are about to be deleted!) Data lifecycle management should reserve data products that it wants to deliver - might require implementing a data lifecycle management interface service (Might want to move data product registration model to also be data-flow-based) Stretch : Implement request to data lifecycle management service to copy/move-in data products Implement either as part of an SDP buffer manager or platform interface ("Helm deployer") deleting objects without reservations For a first version we might especially implement it as a move into a "recycle bin" area instead of an actual "delete" - to make sure we don't lose data Might want to implement some slack (~10 minutes?) to make sure we give other services a chance to add reservations Could especially disable this entirely if - e.g. - the data lifecycle management interface is not currently active
    • 5
    • 5
    • 0
    • PI24 - UNCOVERED

    • AA2 SDP_AA2_services_realtime_pipelines

    Description

      See Feature frame on DP ART board.

      Context

      The SDP buffer is a store managed by SDP to hold data in support of processing. As such, it is expected to hold only objects that are ephemeral in some fashion (with expected lifetimes of eventually ~2-3 days), so we will need a service that monitors and manages it. This especially includes deleting objects - clearly a risky prospect, but required for day-to-day operations of the telescope once we start ingesting significant amounts of data. We need to make sure this is firmly under control, and we make it as transparent and predictable as possible when and where a deletion happens.

      References

      ADR-101 - SDP buffer management and interface to Data Lifecycle Management

      Attachments

        Structure

          Activity

            People

              p.wortmann Wortmann, Peter
              m.ashdown Ashdown, Mark
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Feature Progress

                Story Point Burn-up: (0%)

                Feature Estimate: 5.0

                IssuesStory Points
                To Do00.0
                In Progress   00.0
                Complete00.0
                Total00.0

                Dates

                  Created:
                  Updated:

                  Structure Helper Panel