Uploaded image for project: 'SAFe Program'
  1. SAFe Program
  2. SP-1953

Recoverable Tango Devices in TMC

Details

    • Obs Mgt & Controls
    • Hide

      Devices that can be recovered ensures quicker maintenance and repair steps as the need for re-deployment is removed.

      Show
      Devices that can be recovered ensures quicker maintenance and repair steps as the need for re-deployment is removed.
    • Hide

      Demonstration that a tmc subarray tango device (e.g. sdp/csp sub array leaf and subarray node) in 'State' == FALSE can be re-initialised when the command 'Reset" is given which will cause it to enter the 'state' 'OFF as per currently agreed observation state machine https://developer.skatelescope.org/projects/lmc-base-classes/en/latest/State_Machine.html

      Given the ongoing work with SS-82, demonstration using umbrella charts rather than full SKAMPI is sufficient for acceptance. Future stories/work beyond the scope of this feature to incorporate this work into SKAMPI at a future date will be required

      Show
      Demonstration that a tmc subarray tango device (e.g. sdp/csp sub array leaf and subarray node) in 'State' == FALSE can be re-initialised when the command 'Reset" is given which will cause it to enter the 'state' 'OFF as per currently agreed observation state machine https://developer.skatelescope.org/projects/lmc-base-classes/en/latest/State_Machine.html Given the ongoing work with SS-82 , demonstration using umbrella charts rather than full SKAMPI is sufficient for acceptance. Future stories/work beyond the scope of this feature to incorporate this work into SKAMPI at a future date will be required
    • 2
    • 2
    • 13
    • 6.5
    • Team_NCRA
    • Sprint 3
    • Hide

      1. Simulated scenarios for SubarrayNode for ObsState to be FAULT, and CspSubarrayLeafNode, SdpSubarrayLeafNode for its State to be FAULT.
      2. Observed that SubarrayNode's state does not change to FAULT.
      3. Implemented Reset command on SubarrayNode, SdpSubarrayLeafNode and CspSubarrayLeafNode. As a part of the Reset command, few Attributes values are updated in the Init command for eg. receptorId list, scan duration, scan configuration, etc are now getting reset.
      Also added a mechanism to kill the alive threads wherever applicable.
      MR link for ska-tmc-subarraynode: https://gitlab.com/ska-telescope/ska-tmc-subarraynode-mid/-/merge_requests/25
      4. Created a new taranta dashboard with Reset buttons to demonstrate feature SP-1953.
      5. Implemented integration test case for Reset command with the scenario of bringing subarrayNode into the FAULT Observation state and sdpsubarrayleafnode, cspsubarrayleafnode into FAULT state.
      MR Link: https://gitlab.com/ska-telescope/ska-tmc/-/merge_requests/292
      6. Published docker image for ska-tmc with version 0.8.7 and ska-tmc-subarraynode-mid with version 0.3.8 on CAR .
      Ska-tmc : https://artefact.skatelescope.org/#browse/browse:docker-internal:v2%2Fska-telescope%2Fska-tmc%2Ftags%2F0.8.7
      Ska-tmc-subarraynode-mid : https://artefact.skatelescope.org/#browse/browse:docker-internal:v2%2Fska-telescope%2Fska-tmc-subarraynode-mid%2Ftags%2F0.3.8
      7. Published ska-tmc-mid chart version 0.4.10 containing Reset functionality on CAR.
      Link: https://artefact.skatelescope.org/#browse/browse:helm-internal:ska-tmc-mid%2F0.4.10%2Fska-tmc-mid-0.4.10.tgz
      8. Design of Reset command along with simulated faulty scenarios is depicted in the form of sequence diagram on confluence page:
      Page Link:
      https://confluence.skatelescope.org/display/SE/Resetting+faulted+TMC+Subarray+and+Subarray+leaf+nodes

      Show
      1. Simulated scenarios for SubarrayNode for ObsState to be FAULT, and CspSubarrayLeafNode, SdpSubarrayLeafNode for its State to be FAULT. 2. Observed that SubarrayNode's state does not change to FAULT. 3. Implemented Reset command on SubarrayNode, SdpSubarrayLeafNode and CspSubarrayLeafNode. As a part of the Reset command, few Attributes values are updated in the Init command for eg. receptorId list, scan duration, scan configuration, etc are now getting reset. Also added a mechanism to kill the alive threads wherever applicable. MR link for ska-tmc-subarraynode: https://gitlab.com/ska-telescope/ska-tmc-subarraynode-mid/-/merge_requests/25 4. Created a new taranta dashboard with Reset buttons to demonstrate feature SP-1953 . 5. Implemented integration test case for Reset command with the scenario of bringing subarrayNode into the FAULT Observation state and sdpsubarrayleafnode, cspsubarrayleafnode into FAULT state. MR Link: https://gitlab.com/ska-telescope/ska-tmc/-/merge_requests/292 6. Published docker image for ska-tmc with version 0.8.7 and ska-tmc-subarraynode-mid with version 0.3.8 on CAR . Ska-tmc : https://artefact.skatelescope.org/#browse/browse:docker-internal:v2%2Fska-telescope%2Fska-tmc%2Ftags%2F0.8.7 Ska-tmc-subarraynode-mid : https://artefact.skatelescope.org/#browse/browse:docker-internal:v2%2Fska-telescope%2Fska-tmc-subarraynode-mid%2Ftags%2F0.3.8 7. Published ska-tmc-mid chart version 0.4.10 containing Reset functionality on CAR. Link: https://artefact.skatelescope.org/#browse/browse:helm-internal:ska-tmc-mid%2F0.4.10%2Fska-tmc-mid-0.4.10.tgz 8. Design of Reset command along with simulated faulty scenarios is depicted in the form of sequence diagram on confluence page: Page Link: https://confluence.skatelescope.org/display/SE/Resetting+faulted+TMC+Subarray+and+Subarray+leaf+nodes
    • 12.4
    • Stories Completed, Integrated, Solution Intent Updated, Outcomes Reviewed, NFRS met, Demonstrated, Satisfies Acceptance Criteria, Accepted by FO
    • PI22 - UNCOVERED

    • Cross_Team_Code_Review Team_NCRA

    Description

      {alternative option if implementation not feasiable}

      << update skallop package to restart a tango device being stuck in a state due to defect in code so as to prevent user needing to redeploy entire MVP. The mechanism for implementation will be via commanding the underlying device server to restart the particular tango device>>

      {rescoped to be 'time boxed' within one feature point}

      <<

      Update only TMC subarray nodes and subarray leaf nodes within the available 1 feature point. If time is left then focus on master leaf nodes.

      >>

      Update the implementation of tango devices in TMC so that they can be recovered (by means of re-initializing) once they are in a FAULT state (e.g. dev state == FAULT). Note that the functionality required is for the "attempt" of recovery by means of reinitializing the device (as if it was started up from scratch) excludes the condition of the environment being in an inconsistent state, preventing the device from entering into an operational state.

      Attachments

        Issue Links

          Structure

            Activity

              People

                Adam.Avison Avison, Adam
                g.leroux Le Roux, Gerhard [X] (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Feature Progress

                  Story Point Burn-up: (100.00%)

                  Feature Estimate: 2.0

                  IssuesStory Points
                  To Do00.0
                  In Progress   00.0
                  Complete816.0
                  Total816.0

                  Dates

                    Created:
                    Updated:
                    Resolved:

                    Structure Helper Panel