Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3352

Finish new configuration library etcd backend and integrate

Change Owns to Parent OfsSet start and due date...
    XporterXMLWordPrintable

Details

    • Enabler
    • Must have
    • PI19
    • COM SDP SW
    • None
    • True
    • Data Processing
    • Hide

      See Why? in description

      Show
      See Why? in description
    • Hide

      See What? in description

      Show
      See What? in description
    • 2
    • 2
    • 0
    • Team_ORCA
    • Sprint 5
    • Hide

      Presented at System Demo 19.5 (slides) - the demo used this Jupyter Notebook to run through the performance test.

      AC1:

      AC2:

      AC3:

      Releases of the updated components; the second releases listed contain a bugfix for the Configuration Library (v0.5.3):

      • LMC v0.23.0: REL-868, REL-879 (0.23.1)
      • Processing Controller v0.12.0: REL-871, REL-881 (0.12.1)
      • Helm Deployer v0.13.0: REL-874, REL-882 (0.13.1)
      • Scripting Library v0.6.0: REL-870, REL-883 (0.6.1)
      • Console v0.5.0: REL-872, REL-884 (0.5.1)
      • Data Product Metadata v0.2.0: REL-875, REL-885 (0.2.1)
      • Pointing Offset Pipeline: MR35 updates the version of ska-sdp-config; part of new release version 0.2.0
      • SDP Notebooks: MR26 updates the version of ska-sdp-config
      • SDP Scripts: MR84 updated and released the relevant scripts with the new scripting library (only scripts that can be accessed from SDP by default, i.e. they appear in the scripts.yaml file)

      SDP MR to integrate the components: MR197

      Show
      Presented at System Demo 19.5 ( slides ) - the demo used this Jupyter Notebook to run through the performance test. AC1: ska-sdp-config has been released: REL-867 , REL-880 (0.5.3) This release includes changes from the following changes merged as part of the given MR (only listing this PI's work, for PI18, which is also part of the release, see SPO-2331): fix Transaction delete method ( MR66 ) set up wrapping Transactions ( MR64 ) add a new implementation of the Watcher, including caching queries ( MR50 ) add retry_loop to Transaction and Watcher ( MR70 , MR71 ) implement switching between the old and new etcd backend version ( MR74 ) Testing of the new backend is documented in Confluence: https://confluence.skatelescope.org/display/SE/Testing+new+version+of+config+DB+backend (minor fixes added to MR76 ) For a better installation experience, and to make sure we are not affected unwelcome updates to the client packages, both etcd3-py (old package) and python-etcd3 have been pulled/mirrored into an SKA external repository, which are used by ska-sdp-config: https://gitlab.com/ska-telescope/external/python-etcd3 https://gitlab.com/ska-telescope/external/etcd3-py A new docker image is being set up to be used for testing in CI the components that need etcd: https://gitlab.com/ska-telescope/sdp/ska-sdp-etcd Documentation has been updated: https://developer.skao.int/projects/ska-sdp-config/en/latest/design.html#etcd3-backend-implementation AC2: Performance test of the new backend, showing a clear improvement, is documented in Confluence: https://confluence.skatelescope.org/pages/viewpage.action?pageId=232123744 -- AC3: Releases of the updated components; the second releases listed contain a bugfix for the Configuration Library (v0.5.3): LMC v0.23.0: REL-868 , REL-879 (0.23.1) Processing Controller v0.12.0: REL-871 , REL-881 (0.12.1) Helm Deployer v0.13.0: REL-874 , REL-882 (0.13.1) Scripting Library v0.6.0: REL-870 , REL-883 (0.6.1) Console v0.5.0: REL-872 , REL-884 (0.5.1) Data Product Metadata v0.2.0: REL-875 , REL-885 (0.2.1) Pointing Offset Pipeline: MR35 updates the version of ska-sdp-config; part of new release version 0.2.0 SDP Notebooks: MR26 updates the version of ska-sdp-config SDP Scripts: MR84 updated and released the relevant scripts with the new scripting library (only scripts that can be accessed from SDP by default, i.e. they appear in the scripts.yaml file) SDP MR to integrate the components: MR197
    • 22.3
    • Stories Completed, Integrated, Solution Intent Updated, BDD Testing Passes (no errors), Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI25 - UNCOVERED

    • SOL-G5

    Description

      Finish the implementation of the new configuration library etcd backend started in SP-3164 by adding the watcher functionality. Test the new backend with the SDP and demonstrate that it resolves the performance problems.

      Who?

      • SDP developers and users

      What?

      • A functionally complete implementation of the new etcd3 backend
      • Demonstration that the new backend resolves the performance problems (so we can close SKB-180)
      • Releases of the SDP and its processing scripts using the new backend

      Why?

      • Client package used by the original configuration library etcd backend is no longer maintained and has significant performance issues (see SKB-180)

      Attachments

        Issue Links

          Structure

            Activity

              People

                p.wortmann Wortmann, Peter
                m.ashdown Ashdown, Mark
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete1127.0
                  Total1127.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel