Independently report the TMC operational state and telescope operational state

 This feature is based on the ADR-31 

https://confluence.skatelescope.org/display/SWSI/ADR-31+Telescope+Operational+State+-+definition

The functionality to be implemented was discussed and agreed on 21/06/2021 , see the attached document.

This description need to be updated.

Full implementation of the non-blocking commands and related  progress indicators most probably will will be completed in PI12, when the reference design is available.

To be modifed (as well as ADR-31).

The TMC Central Node monitors and controls all the telescope sub-systems, and based on the information collected can determine the overall status of the Telescope.

TMC operational state:

1. TMC Central Node  attribute state shall report the overall operational state of the TMC* (NOT the overall state of the Telescope).
2. TMC Central Node shall  implement the standard set of commands for the operational state transitions (On(), Off(), Standby()*); these commands trigger TMC operational state transitions.
3. TMC Central Node commands On(), Standby(), and Off() are implemented as software state transitions,  the COTS platform where the TMC (and the rest of MVP) software is running is not affected (it is controlled and monitored via COTS tools). 
4. After the initial deployment and re-start of the TMC software: the TMC software performs initialization, transitions to STANDBY.
5.  After entering state=STANDBY, TMC remains in STANDBY until a command is received that triggers a state transition (On() or Off()).
6. When operationalState=STANDBY TMC:

• monitors the status of other sub-systems and reports status as usual (monitor points, events, alarms are reported as in ON),
• all sub-arrays are OFF and cannot be turned ON (command is rejected).
• Command On() triggers TMC transition to ON; at this time this is purely software transition (does not involve starting up hardware). (TMC transition to ON does not trigger subarray transition to ON; sub-arrays remain OFF (can be set to ON one at a time using command subarray.On())

7. When TMC operationalState=ON:

the command Standby() triggers transition to TMC operationalState=STANDBY, at this time command Standby() is implemented as purely software transition (does not affect hardware). As result of this transition any on-going observations are aborted, resources are released and sub-arrays transition to operationalState=OFF, observingState=EMPTY.

8. Command Off() shuts-down TMC*, at this time (PI11) implemented as a software shut-down (hardware is not affected).

• TBC: When command Standby() or Off() is received should TMC Central Node issue command Abort() to sub-arrays in configuring and/or scanning; this would may be desirable,  but would slow down transition to STANDBY and OFF.

*For Discussion: Does TMC Central Node attribute state report the TMC operational state or only the operational state of the TMC Central Node.  If the latter is the case, which component implements the TMC master control?   

**A need for TMC to implement STANDBY state (low power state) will be discussed with the TDT team; changes may require an ECP.

Telescope operational state:

SKA Telescopes are required to implement a low-power state where the telescope uses less then 5% of the overall power allocation, this has been implemented as STANDBY state.

The telescope state should be reported as ON as soon as the telescope uses more than 5% of nominal power (and therefore is not in STANDBY).

Additional health indicator(s) will be used to detect whether the telescope is ready for for a particular observation or other task. It is to expect that maintenance and updates will be way of life and therefore some components will always be OFF-LINE, in MAINTENANCE, OFF or in FAULT.

At this time,  TMC is software sub-system deployed in the virtualized environment (containers); current version of TMC does not monitor and control the computer platform where it is deployed.

At this time, for testing purposes it is useful to provide a small number of indicators to be used to detect whether the 'telescope' (i.e. MVP) is ready for the next test:

• telescopeState                         -  the overall operational state of the telescope (values: STANDBY, ON).
• desiredTelescopeState    -  the last commanded state. (During TMC initialization set to STANDBY. Note: If after initialization the TMC detects that other sub-systems are ON, TMC shall not trigger transition of other sub-systems to STANDBY automatically.)
• commandInProgress             -  none or the command name
• commandProgress                 -  percent complete, stage or other indication (TBD)

1. TMC Central Node attribute telescopeState shall be used to report the overall state of the Telescope.  This indicator is particularly useful for automatic test execution in the absence of real hardware (i.e. when MVP software is running in the simulated environment and all the sub-systems and components are expected to perform transitions promptly).  In the real telescope, in particularly during commissioning, it will be often the case that some components do not perform transition as expected; and it will be challenging to derive and report the overall state/status.  The usefulness of the attribute telescopeState will be tested over time. Initially we will define a set of values to support automatic test execution.
2. The TMC Central Node implements commands TelescopeOn(), TelescopeOff(), TelescopeStandby() which trigger the Telescope operational state transitions, and are forwarded to DISH/MCCS, CSP and SDP (and other sub-systems as applicable).  Details of the initial implementation of these commands and the values reported by the attribute telescopeState need to be refined and agreed before implementation can commence (i.e. by the beginning of PI, preferable before the planning event).  Details of implementation may evolve over time, as sub-systems and functions are added to the MVP.
3. TBD: OET operational state transitions (how does OET report operational state and how it is affected by the commands TelescopeOn(), TelescopeOff(), TelescopeStandby()).

Note: Once the equipment is installed at the site, setting the Telescope in STANDBY will be a very rare event. In virtualization environment and during early stages of testing in PSI and ITF  re-starting TMC and the whole 'telescope' will occur relatively often.  Implementation should be able to handle state transitions in all environments. implementation of different modes of operation to accommodate different environments will be considered.