Alarm Reporting and Logging Functionality

Placeholder to capture this requirement as identified by s.vrcic during the PO Sync of 3 Jul 19.

The SKA Control System Guidelines define the architectural concepts and technology choices both for alarm handling and logging. The prototyping effort should include implementation of the alarm handling and logging (as defined in the SKA CS Guidelines), so that those concepts can be evaluated and adjusted as needed. Additional benefit is that all software developed and on-boarded during the Bridging period will  conform to the same standards. 

The objectives: 

1. Provide infrastructure/platform to be used by all the teams developing evolution prototype software. 
2. All teams developing evolutionary prototype adhere to SKA CS Guidelines or other architectural and technology choices defined as result of step 1.
3. Evaluate the Quality Attributes (robustness, performance, ease of use, etc) of the proposed architecture and chosen technologies.

A brief overview of the architectural concepts related to alarm handling:

The SKA project adheres to the IEC 62682 Alarm Management standards for the definition of alarms; as per the IEC 62682 definition, an alarm always requires a timely and direct operator intervention. TANGO Control System, on the contrary, uses a more broad definition of alarm. To clearly identify the difference between the TANGO alarm messages and IEC 62682 alarm, the SKA Control System Guidelines define three concepts, as follows:

1. A TANGO attribute alarm is generated directly by the TANGO Core when the quality factor of an attribute transits from VALID to ALARM. When an attribute value is outside the programmed thresholds, the TANGO device state transits to ALARM and a TANGO alarm message is generated and sent to all registered clients.
2. An Element Alert is a higher level notification that happens when a logical combination (alarm rule) of TANGO attribute alarms, attribute values, quality factors and devices states over one or multiple devices, is satisfied. Each Element Alert is exposed as a specifically named and formatted attribute on each Element Alarm Handler device.
3. A SKA Alarm is any Element Alert or combination of Element Alerts that determines a condition requiring a direct action from the operator. The (TM provided) Central Alarm Handler has the responsibility to define the rule-based formulae to identify the SKA Alarms (or Telescope alarms) that comply with the IEC 62682 Alarm Management.

Furthermore, the  SKA Control System Guidelines, define the hierarchy of Alarm Handlers, all based on the Elettra implementation of the Alarm Handler, so that a TANGO attribute alarm can be evaluated at several levels of monitor and control hierarchy, resulting in the status update of equipment and sub-arrays  that include or use the component that generated original alarm.

The Telescope Central Alarm Handler registers to receive the attribute alarms and Element Alerts and based on information provided by other telescope sub-systems determines whether an IEC 62682 alarm should be generated. The attribute alarms and Element Alerts which do not require an operator action, are displayed and logged by TM, as required, so that they can be inspected by the operations personnel  either in real-time or periodically.

A brief overview of the architectural concepts related to logging:

 SKA Logging will be based on two services:

1. TANGO Logging Service (TLS) to support viewing of logs in an Element and centrally for logs across Elements.
2. rsyslog with ELK stack to store logs.

Use cases for ELK vs TLS logs: The central ELK log store will be used for fault finding and forensics while the TLS will mainly be used for "in-time" monitoring of log messages.

Logging targets for TANGO Device Servers: Each SKA TANGO device will log to 3 pre-configured logging targets:

1. ElementLogger log target at default INFO level
2. TelescopeLogger log target at default WARNING level for all Elements
3. Storage/syslog local server at default INFO or DEBUG level

 SKA will use the default logging format defined in the Fundamental SKA Standards (Syslog, RFC 5424). 

The SKA CS Guidelines define the Logging requirements for the SKA TANGO base class.