Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-3921

Implement release and deployment mechanisms for the SDP

Details

    • Enabler
    • Must have
    • PI21
    • COM SDP SW
    • None
    • Data Processing
    • Hide

      See Why? in description

      Show
      See Why? in description
    • Hide

      See What? in description

      Show
      See What? in description
    • 1
    • 1
    • 0
    • Team_NALEDI, Team_ORCA, Team_YANDA
    • Sprint 5
    • Hide

      AC1 and AC2 (ORCA):

      • We generated the EDA configuration for SDP, which is saved in the "data" folder within the chart. This was first released with SDP 0.18.1 (REL-1180). 
      • The Taranta dashboard file, which was already part of the ska-sdp-integration repository, has been also moved to the data directory within the chart, and is part of SDP 0.18.1.
      • As such, one can obtain the files by downloading the chart's tgz file and extracting it. In addition, they are linked to the REL ticket in the description, under "Ancillary products". This method was agreed to with FO and Capability owners.

      AC3 (YANDA)

      It is clear that the current deployment at the ITF does not use the CI scheme. It uses the umbrella chart as presented in https://gitlab.com/ska-telescope/aiv/ska-low-itf.git

      Indeed after a number of test deployments in Azure I found that the only way to deploy the SDP into a running deployment was to "redeploy" the umbrella chart. AIV confirmed this in a slack conversation.

       

      From Alex Hill (AIV)

      So the way we work, updating a subcomponent means updating our Helm chart’s dependencies and values if necessary, and just doing a Helm upgrade - letting Helm take care of tearing down what needs to be replaced etc. In that sense we don’t ever deploy a component into a larger running deployment, we just upgrade the larger deployment.It’s a kind of tricky problem though. One-Helm-chart-to-rule-them-all will be OK when the software is stable. But for now, we frequently need to fix bugs and then deploy the development versions, and it’s a bit of a nightmare to do that when you want to bump e.g. ska-low-cbf-proc, which is a dependency of ska-low-csp, which is a dependency of our ska-low-telescope chart - at leach level of the hierarchy starting at the bottom, you have to build a new chart in the pipeline, then update the dependencies in the level above. I’m not sure what the solution is but it makes me feel like maybe we’re doing microservices wrong. Suggestions welcome…(It’s possible to just override the image that a sub-chart uses through Helm values, but if the newer version contains any changes to the chart itself, then that’s no good…)

      As far as the latest SDP is concerned. G.Hodosan has noted that a latest umbrella chart that works is here: https://gitlab.com/ska-telescope/ska-jupyter-scripting/-/merge_requests/79

      Perhaps subsequent features need to determine a more robust way to deal with the dependencies is required.

      Show
      AC1 and AC2 (ORCA): We generated the  EDA configuration for SDP , which is saved in the "data" folder within the chart. This was first released with SDP 0.18.1 ( REL-1180 ).  The  Taranta dashboard file , which was already part of the ska-sdp-integration repository, has been also moved to the data directory within the chart, and is part of SDP 0.18.1. As such, one can obtain the files by downloading the chart's tgz file and extracting it. In addition, they are linked to the REL ticket in the description, under "Ancillary products". This method was agreed to with FO and Capability owners. AC3 (YANDA) It is clear that the current deployment at the ITF does not use the CI scheme. It uses the umbrella chart as presented in  https://gitlab.com/ska-telescope/aiv/ska-low-itf.git Indeed after a number of test deployments in Azure I found that the only way to deploy the SDP into a running deployment was to "redeploy" the umbrella chart. AIV confirmed this in a slack conversation.   From Alex Hill (AIV) So the way we work, updating a subcomponent means updating our Helm chart’s dependencies and values if necessary, and just doing a Helm upgrade - letting Helm take care of tearing down what needs to be replaced etc. In that sense we don’t ever deploy a component into a larger running deployment, we just upgrade the larger deployment.It’s a kind of tricky problem though. One-Helm-chart-to-rule-them-all will be OK when the software is stable. But for now, we frequently need to fix bugs and then deploy the development versions, and it’s a bit of a nightmare to do that when you want to bump e.g. ska-low-cbf-proc, which is a dependency of ska-low-csp, which is a dependency of our ska-low-telescope chart - at leach level of the hierarchy starting at the bottom, you have to build a new chart in the pipeline, then update the dependencies in the level above. I’m not sure what the solution is but it makes me feel like maybe we’re doing microservices wrong. Suggestions welcome…(It’s possible to just override the image that a sub-chart uses through Helm values, but if the newer version contains any changes to the chart itself, then that’s no good…) As far as the latest SDP is concerned.  G.Hodosan  has noted that a latest umbrella chart that works is here:  https://gitlab.com/ska-telescope/ska-jupyter-scripting/-/merge_requests/79 Perhaps subsequent features need to determine a more robust way to deal with the dependencies is required.
    • 21.6
    • Stories Completed, Outcomes Reviewed, Satisfies Acceptance Criteria, Accepted by FO
    • PI22 - UNCOVERED

    • Low G3 Mid G3

    Description

      The standard release and deployment mechanisms developed as part of Com-C3 (SS-117) need to be implemented for the SDP.

      Who? (Beneficiaries)

      • SDP developers.
      • AIV engineers.

      Why? (Benefit hypothesis)

      • SDP implements standard mechanisms for delivering ancillary software products and deploying the sub-system at the ITFs.
      • Enables easier deployment and testing of the SDP at the ITFs and fosters collaboration between SDP developers and AIV teams.

      What? (Acceptance criteria)

      • An EDA configuration has been created for the SDP. (ORCA)
      • SDP releases are accompanied by ancillary products (EDA configuration, Taranta dashboards) packaged in the standard way. (ORCA)
      • The SDP is deployable at the ITFs using the standard mechanisms (in the SDP integration pipeline, TBC). (YANDA)

      Attachments

        Issue Links

          Structure

            Activity

              People

                m.ashdown Ashdown, Mark
                m.ashdown Ashdown, Mark
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 1.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete917.0
                  Total917.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel