TMC handles sub-system and command failures correctly.

The purpose of this feature is to analyze failure scenarios and agree what is the desired TMC behavior and action when an error or failure is identified.
When the agreed functional changes are implemented TMC will be more robust and will report the reason/cause for the failure to the clients.

• The list of scenarios is as complete as possible at this time, and the desired behaviour is defined for each scenario.
• The patterns of the TMC error and failure handling are identified and agreed.
• The scenarios and desired behaviour is documented and released for review by interested parties (FO, PM, OET, etc).
• TMC components implement error and failure handling as documented. This is verified using integration tests on the ska-tmc-integration repository using mock devices. 
• Stretch: Possibility of implementing BDD scenarios will be considered.

TMC negative scenarios are finalised and documented
google spreadsheet:https://docs.google.com/spreadsheets/d/1XbNb8We7fK-EhmOcw3S-h0V_Pu-WAfPTkEd13MSmIns/edit#gid=0

The Finalised approaches for TMC negative scenarios handling are documented
The approaches are reviewed by subsystem teams (SDP, CSP, Dish and MCCS). 
The finalised approach is documented at: https://confluence.skatelescope.org/display/SWSI/TMC+handling+of+sub-system+failures

Implemented the finalized handling for AssignResources  and Configure functionality on TMC nodes . The failure handling approaches are implemented on affected TMC nodes i.e. CentralNode, SubarrayNode and leaf nodes
The verification of failure scenarios has been done on ska-tmc-integration repository via a set of newly developed tests
AssignResources failure, below are the scenarios considered:
1.TMC SubarrayNode handles the failure when the AssignResources command fails on CSP and SDP Subarrays
2.TMC SubarrayNode handles the failure when the AssignResources command fails on CSP Subarray
3.TMC behavior when Csp Subarray is stuck in obsState RESOURCING
4.TMC SubarrayNode handles the failure when the AssignResources command fails on SDP Subarray
5.TMC behavior when Sdp Subarray is stuck in obsState RESOURCING
6.TMC SubarrayNode handles the failure when the Incremental AssignResources command fails on CSP and SDP Subarrays
7.TMC behavior when incremental AssignResources on CSP Subarray raises exception
8.TMC behavior when Csp Subarray is stuck in obsState RESOURCING after incremental AssignResources
9.TMC behavior when incremental AssignResources on SDP Subarray raises exception
10.TMC behavior when Sdp Subarray is stuck in obsState RESOURCING after incremental AssignResources

MR links: 
https://gitlab.com/ska-telescope/ska-tmc/ska-tmc-integration/-/merge_requests/122
https://gitlab.com/ska-telescope/ska-tmc/ska-tmc-integration/-/merge_requests/126

For the Configure command failure, below scenarios are considered:
1.TMC behavior when Csp Subarray is stuck in obsState CONFIGURING
2.TMC behavior when CSP and SDP Subarray raises exception for Configure command
MR link:
https://gitlab.com/ska-telescope/ska-tmc/ska-tmc-integration/-/merge_requests/124

Observation: For the below mentioned scenarios, tests are implemented and passing when executed individually. 
But for combined runs, random failures are observed hence the tests are marked as skipped, these will be worked on in PI21
1.TMC behavior when Csp Subarray Configure raises exception
2.TMC behavior when SDP Subarray Configure raises exception
3.TMC behavior when Sdp Subarray is stuck in obsState CONFIGURING